As a solvent both of them will strip the oils from your hands. Should you have a sensitivity to them, you'll get some irritation and a bit of redness. If you use it a lot, you'll dry your skin to the point where you get cracking and flaking. Be aware that rubber gloves will slow the exposure, but will become saturated themselves and you will still be exposed. So change your gloves frequently if you do have a sensitivity.
Also, there will be some benzene present in toluene or xylene. Benzene has been shown to cause cancer and interfere with reproduction in males. The OSHA permissible exposure level for benzene is 1 ppm on a time weighted average. (Acetone for example has a 1000 ppm PEL, xylene 100 ppm and toluene 200 ppm) You should work in a well ventilated area and not store the containers in your living area.

The flammability concerns probably trump any exposure concerns for these. You should be fine if you are careful with them.