but if you apply the same rate to the homes that average people have to buy you're going to end up in a shitty spot

And that's why you don't do that and instead make progressive taxation a thing.

If taxing the rich is the goal we shouldn't be talking about property taxes on single family homes unless it's specifically related to second and third homes.

Nah. It is good and correct to tax extremely large/valuable single-family homes at high rates even if they're primary residences.

(Of course, another aspect of the issue is that single-family houses in very high-demand areas should lose their zoning protection so they can be bought out and replaced with multifamily buildings. Reasonably-sized single-family houses should never have gotten to unaffordable valuations in the first place.)