I think the only time I put my off hand in my pocket while fishing is when it is cold out.

D