At risk of getting more technical, some near-future combination of bitnet-like ternary models, less-autoregressive architectures, taking advantage of sparsity, and models not being so stupidly general-purpose will bring inference costs down dramatically. Like, a watt or two on your phone dramatically. AI energy cost is a meme perpetuated by Altman so people will give him money. Kinda like a NFT scheme.

...In other words, it's really not that big a deal. Like, a drop in the bucket compared to global metal production or something.

The cost of training a model in the first place is more complex (and really wasteful at some 'money is no object' outfits like OpenAI or X), but also potentially very cheap. As examples, Deepseek and Flux were trained with very little electricity. So was Cerebras's example model.