Downvotes usually signify "you posted something rude, objectively incorrect, off-topic, or instance/community rule-breaking" and people who think they did not do anything wrong might understandably be frustrated when they are getting the signal they did something like that, but they can't figure out what bad behavior they're getting downvoted for at all. They might want to know what the downvoter thinks they are doing that is inappropriate so they can stop it, or if it is just a troll trying to make people unhappy and thus ignorable. And given the "lol why do you even care, are you that sensitive over internet points" kind of view, it makes it even more annoying because asking what they did wrong might just get them a response like that and no further clue to what offensive behavior they've committed. I know some people use downvote for a mere disagreement as well as for actual objectionable behavior, and trolls exist so downvotes are often safe to ignore, but I'd imagine a reasonable person who does not care about internet points might start getting a little "oh god what did I do wrong" if it's more half-up half-down or mostly downvoted, especially since I usually see rude/spammy/incorrect things downvoted more heavily than things politely pushing against the grain of popular opinion.

Also somewhat relevant: I subscribe to some really small magazines/communities. Some of them are full of posts that might appear bad or controversial by vote count (think 3 upvotes vs 3 downvotes, 4 upvotes vs 3 downvotes), but the posts are entirely on topic and are not particularly biased or incorrect. I feel bad for them because it makes that post look low-quality or like they are about a controversial topic when it is really just a tiny community that probably attracted a couple trolls but not enough legit users to outvote them. I can see that getting the community creator a little upset, having their regular content look to onlookers as if it's trash when it's normal content, and lowering their hopes of attracting people to the community. Although hopefully people can recognize that at that small a number of people it might just be trolls and not low-quality content.