Jim;
My fellow "Raunchy" member in the NBOF, is right. "Grinding up the chaff won't hurt your brew".
However,with some of the South and Central American beans I use, I've found they can contain a lot more residue chaff than others. Too much chaff, when grinding in my burr grinder, would sometimes get sort of "paste like" and clog up the burrs.
Before I bought my first roaster, and still did it "chuck wagon style", I would toss the beans in a colander to not only cool them down but the tossing also brought the lighter weight chaff to the top of the beans, where I'd then use either my hand held or shop vac to vacuum the chaff off the top.
Don't worry! It's easy to "get the right distance", between sucking up the chaff and not your beans.
When I did 2-4 pound batches, I'd set a small table fan up, "OUTSIDE!!!", then standing off to one side, I'd toss the beans in the colander and let my "artificial wind" blow away the chaff.

Also, if you haven't tried them yet, you might give "Sweet Maria's" at "Sweet Marias-dot-com" a shot for pricing and shipping costs. Great folks and very easy to deal with.