This is going to be a controversial take, but I believe this confusion comes from applying the wrong word to the concept being conveyed. The word "privelege" means to grant an advantage/immunity to an individual above the usual rights/advangates people get. I acknowledge that language evolves and the word privelege can evolve to mean a lack of impediments that some people suffer under. But that sort of evolution usually takes generations.

There's a perspective matter at play as well: If the "baseline" rights and advantages of "usual" are somewhere below what "joe average" white man gets, then Joe isn't average any more. And that's a perspective shift we need the whole population to acknowledge.

Personally, I'm fine with acknowledging that I don't suffer the impediments of race or gender that many people do. I suffered under different impediments though: As a kid I was very small, not really having my teenage growth spurt until after I left school. I was also poor. These impediments, while not related to race or gender were no less real. Growing up, I sure didn't feel priveleged. Does a gay, financially secure black girl feel priveleged? I recognise that today, I am priveleged the way that term is applied in modern discourse. I am also neither short, nor poor any longer. But for all that, I still feel like "joe average".

I can see how telling a white man who is burdened with some sort of impediment that he's "priveleged" because he doesn't suffer under the impediments you suffer is going to be a hard-sell. I believe we'd all be better served with a different word to convey this concept. We already acknowledge that the current term of "privelege" is misinterpreted and misunderstood. I am not smart nor connected enough to come up with a new word and spread it, though.