Mike, I fish a lot like you describe and when I do, I never wear a vest. I use a flyfishing shirt. Mine are made by Magellan. They are a Columbia knock off. They have the big pockets on both sides that hold 2 fly boxes and 2 smaller pockets under the big pockets that hold tippet and leaders. I attach my hemostats to the little tab attached to the shirt for that purpose. I have a homemade lanyard (read that large braided line that catfishermen use around here with a knot tied in the end) that has on it clippers, fold up Wal Mart scizzors, 2 sizes of safety pins for cleaning hook eyes and picking out wind knots and a hook sharpener.
I probably have 10 of these shirts in various colors. Mostly neutral colors to match the background I'm fishing. You can get them at Academy Sports. They cost about $20.00 each and they are very durable and comfortable. I wear them when not flyfishing too. I like to make a statement, you know. I like the long sleeved variety. You can roll them up and button the tab to make them a short sleeve.