I don't think it's ever too early to expose a kid to fishing (fly or otherwise), but 4 is a little young to expect much in terms of results. The trick is to get them to the water, then let their interest build as their attention span does. 4 is a great age to get a kid interested in the buggier side-- watching for bugs, learning to identify them and so forth. That and throwing rocks in the water, but if you get them to the water as soon as possible, the rest will come. The most important thing though is not to force it.