The opinion of strangers should have some impact on you. Too much and too little both lead to bad places. A little consideration for the opinions of strangers encourages pro social behavior. And yes choosing to avoid accidentally signaling something is part of pro social behavior.

Personally I like strangers to be less on guard with me, it makes my day a little bit more pleasant. I’m not going to judge people for choosing otherwise with something like a hat that happens to be red, but it’s not a choice I’d personally make