I was always taught that the towel's final role is to abrade and collect the dirt/grime/skin that has been loosened by showering, but that wasn't washed away (which iirc is mostly just the skin and oily grime, not dirt or other large particulates).

This is nasty to me. Do you not scrub yourself in the shower? Like, if you have dirty hands, and wash them in the sink, you can dry your hands on a white towel and it should not get dirty. Same for your body except you usually have more scrubbing tools to exfoliate your skin and remove the dirt so you should be even cleaner come dry time.

I don’t know that I’d call my used towel “clean” per se, but it’s certainly not dirty. It’s not doing any cleaning or removal of anything, just patting up water.

You gotta use washcloths or scrubbers on your body! In conjuction with soap, that will help remove dead skin, dirt, sweat, etc.