As a person with multiple allergies, including the peanut and nut throat closing death kind - I'd caution you that it's quite hard to tell about allergies based on other's peoples experiences. My peanut allergy only happens if I eat them, but I know other people who get reactions even if they only smell peanuts!

I'd probably be real careful on your first few trips. Maybe get some gloves for the handling part. Maybe fish at a park close to the parking lot so you can do an epi-pen and drive the emergency room real quick if you have to.