If you go directly from stream to stream without the boots drying, you're correct. Drying, however, kills didymo (unlike whirling disease spores), so those areas will be ok if you just dry the boots. The problem with felt soles is that they stay wet inside for weeks, making it much more likely that you're going to transport didymo for the stream you fished this weekend to the one you're going to fish next weekend.