You could make the saame argument against commenting, say the wrong thing and you get harassed. But what do you do with someone harassing you? You block them. What do you do when someone is harassing you by downvoting you? You most likely suck it up because you don't know who is downvoting you.

I would argue it's more a tool against harassment than a tool for harassment, especially since anyone determined enough can already access that information.