I think perhaps you ae rubbing it too much with the balled up cloth. I also use Micron pens. I usually let the lettering dry for a day and other hit it with a blow dryer to help set it. Like JKlett I 0000 steel wool first. and lightly brush on the varnish. If you are using the balled up cloth then I would do this. rough the surface, letter, hit with some heat, let dry 12 hrs, brush varnish over just the lettering, let that dry, then apply the rest of your finish.