No one is saying that individuals of any ethnicity can't be bigoted. The point being made is that, in our current society, the societal structure that we call "racism" - the social machinery that empowers white bigots to enact violence against people of color and get away with it, and disadvantages non-white people in all sorts of other ways - lends a certain weight to the prejudices of white folks that just isn't true for the prejudices of other people. Disadvantaged people who hate all white people just aren't a threat to white people the way that even well meaning white people are to people of color.
In other words: the prejudices of those in power are a much more pressing social ill because the ruling majority ethnicity/culture has the power to actively cause harm via those prejudices.