Microsoft tried to make Windows Server popular. Apple sold a server OS and even its own rack-mount servers for a while.

The people using servers, and often the people making the decisions about what to use have a high degree of technical knowledge and skill. The things that drive popularity in consumer operating systems such as being preloaded on devices and having a polished GUI don't have as big an influence on experts.

Customizability, reliability, and performance do have a big influence on what experts choose, and Linux wins on those points. There's also the history of proprietary Unix being big in the server/supercomputer market, and Linux is an obvious successor.

A big piece is licensing. When you're throwing hundreds or thousands of processor cores into a data center, somewhere a Microsoft VAR is just drooling to sell you datacenter license packs that you'll need to renew/repurchase for every major OS upgrade. Ah, and you'll need device/user cals. Oh you want to manage it too? Oof.

Microsoft comes out with Windows Supercomputer Pro.

They sell 6 copies.

When you're running exotic hardware, everything is custom. Linux is the most easily customizable.

One thing I'm not seeing mentioned is Windows forces restarts and updates, which admins really don't want to have servers interrupted by, at intervals of Microsoft's choosing. With Linux you choose 100% of the restart and update times.

You answered your own question:

That’s astonishing in a capitalist world where absolutely everything is commodified.

If you bought windows for the hundred or so servers you manage how much is that going to cost you up front and then again every time Microsoft decides to just end support and roll out a new OS? How much work does it take to manage those licenses and make those updates? And what are you getting in return for that additional expense?

Linux has an extremely flexible architecture. Before Linux, most servers ran on UNIX, and before that, well, networking was in a very early and rudimentary stage.

When UNIX licensing shenanigans kept happening, Linux was a more and more attractive option as it matured.

Today, Linux is an incredibly flexible, reliable and performant OS. It's free, in most cases. Why would anyone use anything else? HPC software all runs on Linux and UNIX. You can run it on a tiny little SBC like a raspberry pi, you can run it in an embedded system like car infotainment or a smart meter, and you can run it on ultra high-performance supercomputer clusters. It doesn't give a damn; it just works.

Why would we use anything else? Apple's ecosystem, while great, makes no sense in the server world. They have their own unique directory service that nobody wants to support (unless they're trying to sell something to Apple themselves), they have total control over the OS and its capabilities, and it's technically illegal to modify. Windows has a heavy GUI, and its command-line interface is middling at best and difficult to learn. Windows excels in backwards compatibility and ease of deployment, which makes it ideal for small and medium businesses, but it quickly becomes irrelevant once you scale to a certain point. This is why they've got their Azure AD product, for example. It's attempting to fix the scalability issues with Windows Server. Having spoken to some of the developers of Windows Server, it's also plain as day that Microsoft is only really maintaining Windows Server to collect on their existing contracts. They have no desire to grow that part of their business.

With all of this in mind, Linux the most obvious choice. It takes no time at all to slap a copy of Ubuntu Server on a pizza box and have a functioning server up in an hour. Everything else is more complex, slower, and costs money.

It just works better.

Most servers around the world run Linux.

True.

Why can’t these big tech companies manage to sell their own software to server operators

Lots of companies large and small are running commercial distributions of Linux including with paid licensing for products Like RHEL (Red Hat Enterprise Linux) or SUSE. Millions of other servers are on co-branded versions of Linux that are provided for free to the customer as long as the customer continues to use the company's service. Examples here are Amazon Linux or Oracle Linux (both of which you're only allowed to run if they're operating on Amazon or Oracle Cloud servers. Now, these same companies likely also use unlicensed free Linux in places, either disposable clusters or labs, but if an application has commercial uptime requirements (meaning downtime costs money), few companies run free Linux in those specific applications.

or supercomputers?

This is a frighteningly small install base to try to sell a commercial operating system on. How many supercomputers are there in the world? Perhaps 1000? Moreover, these are such specialized set ups that trying to make a one-size-fits-all OS is likely impossible.

Why is an open, free project that is free for users so superior here?

Just because there is free Linux does not mean that all Linux servers in the world are free.

It was mentioned in the threads, but I don't think it got enough attention. Mostly it comes down to money. Yes, customization, efficiency, etc of Linux, but also because every $ not spent on licensing is a $ you can spend.on making the data center better. So maybe it's buying.more hardware, or having more money for infrastructure like electricity , cooling, and bandwidth. Or just plain profit. Licensing is a nightmare with microsoft. Rather save the money and time and.make a better data center.

Because they have to work

Because being beholden to a single mega corp is a massive business risk. If MS decides to yank support or they go under then you suddenly need to migrate to a new OS for your entire server farm with all the problems that come with it, like having to port all your custom software to the new OS. Also if the US decides to sanction the country your business resides in than you are also fucked if you run your entire IT infrastructure on Windows. Linux being opensource mitigates all those problems. Like if for some reason you can't use the distro you use you can always migrate to a fork.

Linux requires a lot more knowledge to get functioning properly, but gives more freedom when you do.

Good thing IT experts aren't regular users and can handle that just fine. Supercomputers datacenters also have their own support on hand. They don't need to outsource it like they do in regular businesses.

That and the licenses.

Windows sucks. Linux has so many more features than you could ever think. Ulimit is a major one. B.P.F. is another.

It works well, has little to no vendor lock in, and has reasonable or no licensing costs.

Also drivers can be developed without needing anyones permission or validation like with Windows drivers.

Most servers do not need a fancy user interface. They don't have monitors attached to them. Most of the other OSes you listed come bundled with a ton of user experience software along with the expectation a human will directly interact with them.

Servers are typically deployed and managed with automation, by the thousands. Lean. Simple. Secure. Tuned for their specific purpose. This is a lot easier to do with Linux than an all purpose general use personal computing platform.

Linux offers near-endless customisation and Kernel is also open sourced for any kind of (performance) tweaks.

Moreover, Linux is, by design, better suited to be a server OS than desktop OS.

These are the same reasons why most of the web servers across world runs on Linux based distros.

Amazon's and Google's OSes already are Linux variants. The thing that makes Android is the GUI userland, not the underlying system which is just Linux with a libc implementation not by GNU.

Us oldsters remember the UNIX wars. There were a number of different flavours of UNIX fighting for commercial domination. In the end they all failed because of fragmentation. This is why Linux won.