I think the species-specific mentality exists due to the move towards at least imitation flies, and even realistic flies when targeting trout. This is precipitated by the thought that trout are very selective feeders.

I was going say that any trout fly will work at some point on the pannies, but not all panfish flies will work for trout, thinking about the gaudy, flourescent colors we sometimes tend to use for warm water quarry.

But I'm going to change my mind already and say that even those gaudy panfish flies will work for trout. Yes, I do believe trout can be more selective in their feeding than panfish. But having recently gotten into tying the classic wet patterns (for panfish), it has become clear that even the trout like flashiness in their flies at times. The classic wets cover the color range from all drab brown in a fly to bright red, yellow, orange, and other colors with plenty of shiny tinsel.

So in short, your boxes may include some slightly more hatch-specific patterns for trout, but certainly (at the risk of offending the purists) the same box absolutely can be used for both warm and cold water situations.