Short answer: use uBlue.

Longer answer:

Even though uBlue is technically "downstream", it also isn't. uBlue builds its' packages automatically, and you are never more than a few hours (1 day max for huge updates) away from upstream. It feels more like "sidestream" (if that word exists?).

One reason it exists is, as you already said, because layering takes quite some time.
At least I personally don't wanna use stock Fedora (Atomic) and would install some codecs, tweaks and such anyway, and uBlue does that already for me.
Update time doesn't matter anymore for me, because uBlue updates itself automatically in the background. Silverblue doesn't do that afaik.

Depending on how "custom" your system should be, you can take a look at the uBlue builder, where you can create your own image based on already existing ones if you like.

The cool thing about Fedora Atomic is, that you don't have to stick to anything. If you don't like something anymore, you can rebase in less than two minutes without any hassle and jump from image to image, no matter if it's an official one (e.g. Silverblue) or some obscure uBlue image.