I honestly don't think there is such a thing as "too young." If they are not outright flyfishing, then exposure is the next best thing so let them tag along.

My brother started at 5, I was about 10. My son and daughter have been at it since 4 and 5 but at age 7 and 8, they are just getting proficient. Both were going fishing/camping though since they were in a carrier. Now, they just love to fish as much as I do and it's all they ever talk about.