The problem then is that by responding, you're engaging with it which typically helps it spread in the algorithms*. Ideally there should be multiple downvote options - maybe separate it out as "misinformation" vs "bad opinion" or something. Removing downvotes and banning users who disagree is the typical cult strategy (recall the classic cult sub, r/thedonald, was notorious for this). If you're worried about downvotes being used to silence people, maybe another way to mitigate that would be a "sort by downvoted" option so that being downvoted a lot could actually put you at the top of at least that feed.
*On Lemmy, notice the following:
Active (default): Calculates a rank based on the score and time of the latest comment, with decay over time
The ancient romans actually didn't have gendered bathrooms. Or separate stalls. Just everything hanging out together, watching each other piss and shit. And then they'd all wipe their butts with a common sponge on a stick.
But then that was before Big Toilet Paper.