Right now i live in South Western Florida fishing in the mangroves around Sanibel island.I used my vest as well but did not carry boxes with flies so it was a lot lighter.
You could try using a sling pack instead of a vest. While not in use it rides on your back.