I did an early june trip a few years back. Firehole was the spot to be. PMDs (throw in a few pink ones and you'll have a blast.
Much of the east side of the park is closed until july for the cutts to spawn, so you can forget the Yellowstone river.
I took a day trip over to the Henry's Fork and had a good time there also.