I'll defend the right for anyone to speak their mind, but I'll allow the masses to take their pound of flesh
This is a contradiction. Something isn't a right if you allow open season for others to actively target and suppress.
Otherwise, Stalinism is also technically "free speech": you can say whatever you want, but there will be consequences.
Be clear about what you mean.
At least you admit to not acting in good faith