Currently it takes ~50 minutes to recompile the kernel. Are there any tutorials what drivers to disable to speed up this process?

Step 1: Buy a faster CPU.

The only thing you could do is ccache but that's just a cache and can get invalidated whenever.

After doing that I will try to compile it with -O3 and LTO.

Don't use -O3, especially when your goal is to harden. It has no measurable benefit beyond measurement bias due to memory layout changes and some of its optimisations may produce wrong code which is a big no-no if your goal is to harden.

install ClaimAV

Are you planning to host a file share for Windows system or what are you trying to achieve using ClamAV?

install flatpak versions for every non open-source app

You're going to such lengths and even consider snake oil in order to "harden" your system and then you're telling me you want to run proprietary (often known malicious) software on it?

What are you trying to achieve here? What do you want to protect against whom? Create a proper threat model before you wildly apply "hardening" that is likely ineffective at protecting against the threats that actually matter to you.

I also want to have SELinux.

Good luck with that. Distros with proper SELinux setups (i.e. Android, Redhat) employ teams of people to write SELinux rules for them.

I won't discourage you from learning SELinux but know that setting up SELinux for your entire system when the distro does not support it already is not something you can realistically achieve on your own.