Phones tend to have more specialized and proprietary hardware, so you can’t just take the standard Linux kernel, use it there, and call it a day.

Eh, you sort of can on some phones, e.g. OnePlus 5 and 6; on some others it's just a couple dozen patches away from working.

But I’d be surprised if the people working on this weren’t aware of that fact, and I hope they are working on abstracting the hardware layers more so that every mobile Linux project doesn’t have to start from scratch every time.

The problem with other phones isn't "abstracting the hardware" (this is done by the Linux kernel), it's reverse-engineering the drivers so that they run on whatever kernel you want and use the open standards required by the "desktop linux" userspace. In fact, if you look at the "supported devices" list for all those mobile Linux distros you'll find a fairly similar set; that's simply all devices for which manufacturer's (or reverse-engineered) drivers are available. It's not like FOSS people are writing drivers specifically for their distro, which wouldn't work with any other - only corporate Android vendors do that!