"Gamers" are also a group one elects to be a member of, while one is categorized into a race, sex or gender from birth. One is elective, the other is descriptive. No one chooses to be black, or white, or born with male or female genitalia, etc, etc. And a lot of negative views are often along the lines of a rare bad thing being more likely performed by a certain demographic being extrapolated to accuse that demographic of being dangerous or harmful in general (usually an out-group, though under some ideologies it's only acceptable to have this view with a target perceived to be the in-group - as regards blame they essentially reverse the perceived in- and out-group roles).

To turn it around on you though, imagine we picked some other elective group (a hobby, a political or ideological leaning, that sort of thing) that you are likely to look positively upon (and maybe even be a member of) and did the same kind of thing. Let's say...feminists? Would it be acceptable to accuse feminism or feminists of anything negative I can point to any group thereof doing, and if you aren't one of the ones who actually does that then you should not take offense, right? Not feel defensive at all, not question or challenge the assertion at all, right?