Aside: WTF are they using SSDs for?

LLM inference in the cloud is basically only done in VRAM. Rarely stale K/V cache is cached in RAM, but new attention architectures should minimize that. Large scale training, contrary to popular belief, is a pretty rare event most data centers and businesses are incapable of.

…So what do they do with so much flash storage!? Is it literally just FOMO server buying?

AFAIK this has already been a problem, you can find Samsung M.2 SSDs for cheaper than Samsung SATA SSDs at the same capacity, because their cloud customers have all flown past classic SATA/SAS for NVME U.2 and U.3, which is much more similar to M.2 due to NVME.

I was planning on adding a big SSD array to my server which has a bunch of external 2.5 SAS slots, but it ended up being cheaper and faster to buy a 4 slot M.2 PCIe card and buy 4 M.2 drives instead.

Putting it on a x16 PCIe slot gives me 4 lanes per drive with bifurication, which gets me the advertised maximum possible speed on PCIe 4.

Whether or not the RAM surge will affect chip production capacity is the real issue. It seems all 3 OEMs could effectively reduce capacity for all other components after slugging billions of dollars into HBM RAM. It wouldn't just be SSDs, anything that relies on the same supply chain could be heavily affected.

This seems like a non issue dramatised for headlines, they are phasing out outdated sata connection to only favour current m.2.

It's like gpu and motherboard manufacturers announcing they are no longer including VGA ports in favour of DVI display port and HDMI. I don't think that was a bad thing.

I'm sure some people who are lucky enough to have hardware that still requires SATA want to keep upgrading to new SATA devices but it's been enough time. I'm ok with just m.2 now.

Samsung makes some of the best SSDs!!

Just bought 2tb for $89.

Syrup of Squill.