I'm sure what I'm about to say has already been echoed by others, but there are a few factors working against them. This is from a US/Canada perspective. Other countries may have more or fewer barriers.

1. Mobile hardware - Mobile hardware has higher security. Some of this is by design for the user, since mobile devices are more likely to fall into wrong hands than e.g. a desktop. Some of it is corps preventing users from using their hardware in ways they don't want you to, though. The level of locked down mobile hardware has only increased over time.
2. Carriers - This one is particular to North America, I think. Carriers here have a long history of meddling in phone hardware - from bloatware to SIM locks and everything in between. One of the things they do since LTE is require device makers to pay them to get certified to make calls on their network. Linux capable devices are too niche to be able to afford this. This is why Fairphone, for example, even with its Android-based OS, only works on one carrier here.

Those are the main two barriers here. Things like apps can have workarounds for those that would be interested in early adoption of Linux phones. But there's no way around the combination of carrier certification costs and limited options of only very locked down hardware.