It inhibits your instance from filtering content at all, and prevents your instance-native content from self filtering before reaching larger feeds.

It also demolishes your ability to natively deal with problem comments without mod or admin intervention, which adds more work to moderation while also making power mods more powerful.

Lemmy is aping the reddit design, which has downvotes built into how the design inherently filters content. Removing the user driven content filter without redesigning the way the site works usually just makes problems. Like if mastodon removed retweets, or something.

The analogy isnt perfect, but its like a go cart without breaks. Normally, anyone can tap the gas or break pedal once each, to speed up or slow down a post or comment. A shitty go cart (both shit as in poor quality and shit as in bad/bigoted/counter community) gets its break slammed by multiple people, and stops in its tracks. Great stuff gets the gas slammed and flies forward.

But without the break, users cant slow anything down, just hope no one else speeds it up or that it gets pulled from the track. This makes brigades easier, absent mods more noticable, and bad faith mods harder to push back against from the community.

These issues only really metasticize if a larger percent of instances block downvotes. But it will hamper content coming out of bahaj if the instance population starts to really grow.