You're obviously allowed to care and to express that. I hope you don't hear me saying that I think you're dumb or anything like that. In fact it's pretty much the exact opposite. Most of the time when I see people complain about downvotes they're trolls upset that it hampers their ability to be a troll but you seem like a reasonable person so I am simply seeking to better understand your perspective.

To put it another way I agree with your first sentence but I don't generally feel like visible downvotes prevents that from happening. I actually think downvotes function to encourage good discussion more often than not. That isn't true in all cases but it seems easier to be discouraged or get the wrong idea about prevailing sentiments if the dumb comments are just sitting there next to the good comments with no indication that a majority of other users also think they're dumb.