I don't think downvotes, as they are now, are for any particular purpose other than showing the poster you don't like the post. What I mean to say is that limiting downvotes for off topic stuff isn't enforceable and poor quality content is pretty subjective. Not sure if you are aware of the bean posts recently, but I didn't like them and would consider them poor quality memes. Almost everyone else loved them though, which is totally fine, but I still downvoted those.

Anyone armed with critical thinking skills will eventually reason themselves out of echo chambers, just like your story proves! Although I feel like the ban was more due to the admins of the sub than the downvotes. There are communities where downvoted content isn't removed if it doesn't break any rules, those are the ones people should look for.