I assume device drivers, but probably it is much more complicated than that

Yes, device drivers are an issue. Reverse engineering them is a bitch and slows you down, particularly if you want to support a wide range of models and those models keep getting hardware updates.

But that's not all, software ecosystem is another big one. Android and iOS have seen two decades of people developing software for them. In order for them to want to port their software over to your cleanSlateOS, it would have to have a significant user base. And in order for cleanSlateOS to draw that significant user base, it would have to have an attractive suite of apps to run on it. It's a catch-22.

You could, in theory, try to develop emulators or compatibility layers so that Android apps will also run on cleanSlateOS. But that, again, is time-consuming, will never be free of friction, and require you to make compromises with regard to security and privacy (many apps simply don't run properly without Google's main piece of spyware, the Play Services). It will also kind of tie you to Google again - and that was the thing you were trying to get away from in the first place...

I have a GNU/Linux phone I carry in my other pocket. Here are the biggest issues I can see:

1. Driver support for components in the mainline kernel (lets you install any distro and things like camera, Bluetooth just work)
2. Power management; turns out it is a hard technical problem to have your phone suspend to save energy, while being awake enough to know what and when to turn back on to receive chats/calls, playback music, etc
3. Cameras have a lot of stuff beyond drivers happening behind the scenes these days in software that would need to be developed, especially given it is a big reason people choose their phones for
4. Phone certification is tough, this has stopped even companies like Fairphone from shipping their devices worldwide, I imagine even harder for a device like the Purism Librem 5 where you can literally upgrade Wi-Fi, BT, and cellular generations like a gameboy cartridge
5. App ecosystems take a while to build up, it is a chicken/egg scenario. I think things are in a useable state for all the default apps an iPhone has, but if you want Uber, Uber Eats, you either have to draw even more power essentially running Android via Waydroid, or use a typically more janky web app that may be missing some features

FSF has a project called LibrePhone