Jeff,
You're describing an early season behavior that I run into all the time with brim and other sunfish. The fish are not yet on a heavy feed and instead roll their body over the fly to sink it. They don't take the fly in their mouth so you can't successfully set the hook. Swatting the fly with their tail is another early season non-feeding trick. When the fish are in this mood you can go the whole afternoon without a good hook up. Strangely enough, the fish don't seem to get hooked in the body either. If you find the solution, be sure to share it with the rest of us. 8T