I used to have those concerns. But I still switched earlier this year from being a die-hard Microsoft user since DOS in the 90's. It's no where nearly as bad as you make it sound.

First of all, the controller not working is only the tip of the iceberg of the bullshit sandwich Microsoft has been serving its users and it has gotten far worse now with Windows 11. Microsoft has turned Windows 11 an anti-consumer nightmare of a platform that has zero care for privacy or even for treating paying users as anything else than a source of additional income to exploit further through things like ads and data mining. On a platform they paid for already, must I say again.

Secondly, you can dual boot which means you don't have to abandon your current setup and always have the option to go back to it should it not work for you. That being said, I haven't booted my Windows partition in months and am increasingly considering repurposing the drive for something else now.

Thirdly, what very little problems I encountered were a simple google search away to be fixed. And I am far from being a superuser in that environment. I tried to use Linux 10 years ago before and it was a PITA and I gave up. It isn't like that anymore. It is much better. Things just work now unless you pick a shitty distro.

Finally, I've had a harder time finding the settings in a Windows machine after an update that moved things around than I ever had when I first used Linux. And with Linux, especially if you use KDE Plasma as a desktop environment, if something isn't where you want it, you can customize it to be exactly how you like it. You can make it mimic Windows if you want. There are even custom themes that make it look exactly the same if you really don't want to change.

And even if you don't mind that rapidly growing list of major irritants, many people including myself cannot even upgrade to Windows 11 unless they buy a whole new machine even if they wanted to because of the arbitrary DRM chip requirements. And they're dropping support for Windows 10 next year. So looking down the barrel of having to pay for a new computer while the current one works perfectly well, plus having to pay for another Windows license with which Microsoft will monetize the shit out of my usage of the platform with zero regards to my privacy, making the jump doesn't sound that bad of a decision anymore. I did it and I'm glad I did.