Agreed, this has been my experience as well. I tried switching to full time Linux multiple times. I had already used it on my laptop for years but on my desktop I kept going back to Windows because things on Windows just worked the way I wanted and thought that for some things there weren't any Linux alternatives.
That was until two years ago I challenged myself to only use Linux for a month. I've been using Linux on my desktop ever since and only use Windows now and then to play a single game that doesn't work on Linux due to anti cheat.
I'd consider an optical drive emulator like GDEMU so you can store your entire collection on an SD card. This ensures your physical collection lasts longer and loading times are shorter as well.