Hmm. For me towels are optional. I air dry. I think that feeling grossed out is a normal reaction because some dead and moist and loose skin comes off when doing a "dry towel scrub".

The same glands in your pubes are in your armpits and hair AFAIK, so if any one gets nasty and stinky bacteria, it will spread. Ideally a nice smelling person isolates the stinkiest of the three and cleans with soap and once all three are clean then don't care?

I think attacking your person " calling selfish" when taking something off limits of yours may be a pattern. I think he should have just gone without (as he would have if you weren't there).

I would confront him. First i would write out the facts, Second starting a convo to agree on the facts and third negotiating out the pattern that you want to play out in similar situations in the future.