There's also the possibility of having genuinely good intent, but still speaking entirely from your own conjecture of what might make others uncomfortable.

Ultimately, you should always talk to the people actually affected and take action based on that. But anyone can and should start the initiative when they think something is harmful.