That's kinda dismissive towards racial, sexual, and various other forms of discrimination people face at a work place. It is possible for everyone else to be an asshole when, say, you're the only black person, woman, disabled person, etc, and everyone thinks less of your work and capabilities independently of how you performed or contributed.