Ive always felt the same, throughout my life I've had dumb asses for friends who would say "oh shit Anon here watches alot of anime" and i feel encouraged to downplay it for fear of being associated with weirdos they may have past experiences with.

Being older now i recognize that would be on them for judging based on a label. I don't hide that I watch anime, but I cant say I don't still feel odd talking about it. Especially because its just like any other show to me, its good or it isn't, I'm not some huge anime person, I just love a good story, i don't care about the medium.