Binary reputation systems aren't good. I can say something that right and it can be downvoted because it goes against a person's beliefs, because I'm unpopular, because a certain group doesn't like it, because, because, because. Popularity is not a good measure of quality. Just look at the "publish or die" system. Just because you've been cited multiple times doesn't make your paper right.

Imagine a trans contributor being downvoted just because they're trans. How is that a good system? Do you expect trans people should only contribute in software projects where trans people are accepted? How are you going to prevent brigading?