The whole point is that people can downvote you for any stupid reason and doesn't accurately reflect why a post is being rejected by the community. People will do it simply because they want to silence you.

And it causes issues with brigading, botnets, etc. It's why hexbear and lemmygrad are being defederated -- and their members are able to get around it simply by making accounts on other servers and going right back to the brigading. Removing the voting option and giving admins tools to IP ban everyone from an instance upon defederation would go a long way toward fixing the problem. Just putting the hurdle of needing a VPN to regain access alone will deter a lot of idiots.