You've probably read at least some of Rick's Panfish articles - but just in case you haven't: Rick is absolutely one of the most persistent fly fishers I've ever known. He has more than one rod loaded with possible flies for that particular day's adventure.
If one doesn't work, he changes. Ah, but that isn't all. He changes HOW he fishes those flies as well. How deep, how fast they sink - or are retrieved, how long he waits between striping the fly in....
In other words, you just haven't tried enough different things to make a fly work.
How fast does a plastic worm sink? How does it look in the water on being retrieved?

Think like a fish - in this case you want to match the hatch, right? Except the 'hatch' is a plastic worm.

Last but not least - have fun with the problem!

Hugs, LF