Like it or not, things sometimes become symbols for other things, and especially given the political climate, it makes sense for people to get their guard up if they see something that looks like one of those symbols that represents, say, an utterly toxic ideology.

Not wanting to make others navigate that stuff unnecessarily is a sign of thoughtfulness and pro-social tendencies, not weakness.

The opinion of strangers should have some impact on you. Too much and too little both lead to bad places. A little consideration for the opinions of strangers encourages pro social behavior. And yes choosing to avoid accidentally signaling something is part of pro social behavior.

Personally I like strangers to be less on guard with me, it makes my day a little bit more pleasant. I’m not going to judge people for choosing otherwise with something like a hat that happens to be red, but it’s not a choice I’d personally make