Your third paragraph is a perfect example of why it's so important to talk about these things intersectionally. Because yup, there is definitely a "tall privilege" for people who are taller, and a "pretty privilege" for people who are more conventionally attractive. And the privilege of wealth and class are the most obvious and intense of all. And anyone can have any combination of different privileges, gaining an advantage or not in any number of hundreds of different aspects of life.

Could there be a better word used? I dunno, maybe. I'm reminded of arguments around things like "defund the police". The argument is that even if the term isn't literally the most accurate, it gets across the broad message in a pithy way that's much easier to spread. In the case of privilege, I believe it may be an academic term that has breached the containment of academia and is being unfortunately used among a public that doesn't have the necessary on-ramping to understand what it means with context. Like what happened to "critical race theory".

It would be wrong to tell someone in the broad sense that they "are privileged". Instead, a person may have formed a specific opinion due to their privilege. To use some examples from this article, a man might not understand how it can feel unsafe to walk at night due to his male privilege. A woman might not understand men's inability to express emotion due to female privilege. For what should be obvious reasons, some kinds of privilege are more significant in more areas of life than others, so the line can get fuzzy when it comes to whether saying someone has privilege is appropriate. But it's still worth paying attention to.