Back in late September I was acting as a lake guide for a Casting For Recovery retreat in South Jersey, the main purpose is to have the lady you're guiding catch some fish. We had caught a couple really small sunfish and as we were wading along the shore we were pushing a large school of small sunfish in front of us. The sunnies were sticking close to the shore. We both had polarized glasses and in front of us about 20 feet from shore we could see a line of large bass with about 10 feet between each fish pointed toward the shore. Every so often one would rush in an blow up the school of sunfish, come back and resume its position in the line. They were so intent on the sunfish that we were able to get within easy casting distance. I don't know how many times I changed her flies but even dragging the fly right in front of their noses got no response. With hindsight I realized the flies were "swimming" in a straight line and the sunfish were swimming in circles like a bait ball. Another time I was up on Pine Creek during the Green Drake hatch, I tried emergers, dries and spinners but none of the rising fish were interested and none of the other folks were having any luck either. I noticed what looked like a fine snow rising off the water, taking a closer look I could see tiny white mayflies floating on the water, a size 20 or smaller. I checked my box and I had three white midge dries in my box. Three casts, three takes, three broken tippets.
Confidence factor is another thing. I don't have a whole lot of confidence in Wooly Buggers. I just don't seem to catch a whole lot of fish on them so there are only a few in my box. Same with the Clouser. I had heard so much about them I tied some up. After a couple of years I gave up on them. Then I watched Bob Clouser tie one and realized I had been tying them wrong. Mine looked like Clousers but didn't have the same action. I've had much better luck with them since I started tying them the "Clouser way" and they now have a place in my box.
I agree with Rick, it depends on the day. If dragonflies or damselflies are hatching you won't catch a lot if you don't have a pattern in your box that imitates the nymph or a dry that mimics a cripple. If the bass are focused on 1 to 2 inch bait fish, you might catch some fishing a 4 or 5 inch pattern but you'll catch far more on a smaller pattern.