I think there was a study a few years ago that found drivers are more careful around people who don't wear as much safety equipment. I can see if I can find it late, but I get the impression that driver perception does change depending how you look.
Although I feel like it's the opposite of your experience. If seen several single-speed/messanger/delivery bikes get cut off aggressively by drivers turning corners. Fewer regular bikes although I've seen some crazy near misses of people riding bike-share bikes.