I think you are on the money. Rubber must be too active in the cold water. It is not natural to the fish which are used to slow moving bait. To confirm this consider that the the BASS guys that fish the "Float and Fly" in cold months use hair jigs instead of rubber skirted jigs. I knew that for at least 10 years, but I probably would not have made the connection with fly fishing unless you pointed it out.