I wouldn't want to be associated with vegans, either, tbh. But if someone shows interest or disdain regarding my eating habits, I tell 'em my take. If they like it, cool. If they scorn me, they can fuck right off.

I'm not vegan, and don't really intend to be. But I strongly prefer to eat humanely raised and killed meat. That's pretty uncommon and expensive, so I often just skip meat.

..if someone's interested, I'll tell them about how I kill and eat animals, and why, as well as more about why I avoid most commercial meats.

But yeah, social stigma. If you want to make change, first you've got to sort out your own feelings on that. Really, that's true for any change, not just socially stigmatized ones.

Edit: consider this fun tidbit. Since Veganism is primarily about consent and how an animal can't give it, the only truly vegan meats are vat meats.. ..or other humans (if they consent and aren't under duress).