It could be if it’s run locally.
If you agents run on your hardware, navigate crappy apps and websites and such for you, what do you need the corporate cloud for? How can they show ads or monetize you through that?
That’s the war raging right now, open-weights vs closed weights.
They aren't specialized though!
There are a lot of directions "AI" could go:
There's all sorts of wild directions the sector could go. Having the flexibility of an ASIC die would be a huge benefit for AMD, as they don't have to 'commit' to any particular direction like Nvidia's monolithic dies. If a new trend takes off, they can take an existing die and swap out the ASIC relatively quickly, without taping out a whole new GPU.
With AMD’s IP, they could make a hybrid chip, eg a (for example) bitnet ASIC hanging off a GPU for flexible, cuda-compatible compute where needed.
Nvidia sorta does this now (with tensor cores being a separate part of the die), but with their history of MCM designs, AMD could take it to an extreme.
One one more thing, I saw you mention context management.
Mistral (24B) models are really bad at long context, but this is not always the case. I find that Qwen 32B and Gemma 27B are solid at 32K (which is a huge body of text) and (with the right backend settings) you can easily run either at 64K with very minimal vram overhead.
Specifically, run Gemma with the latest llama.cpp server and comment (where it will automatically use sliding window attention as of like yesterday), or Qwen (and most other models) with exllamav2 or exllamav3, which quantizes the kv cache down to Q4 very efficiently.
This way you don’t need to manage context: you can feed the LLM the whole adventure so it doesn’t forget anything, and streaming responses will be instance since it’s always cached.
Oh, one thing about ST specifically: its default sampling presets are catastrophic last I checked. Like, they’re designed for ancient models, and while I have nothing against the UI it is kinda from a different era.
For Gemma and Qwen, I’ve been using like 0.2-0.7 temp, at least 0.05 MinP, 1.01 rep penalty (not something insane like 1.1) and maybe 0.3-ish dry, though like you said dry/xtc can really mess up some tasks.
Also, another suggestion would be to be careful with your sampling. Use a low temperature and high MinP for queries involving rules, higher temperature (+ samplers like DRY) when you're trying to tease out interesting ideas.
I would even suggest an alt front end like mikupad that exposes token probabilities, so you can go to any point in the reply and look through every “idea” the LLM had internally (and regen from that point of you wish”). It’s also good for debugging sampling issues when you have an incorrect answer (as sometimes the LLM gets it right, but bad sampling parameters choose a bad answer).
As long as it supports network inference between machines with heterogeneous cards, it would work for what I have in mind.
It probably doesn’t, heh, especially non Nvidia cards. But the middle layer may work with some generic OpenAI backend like the llama.cpp server.
Late to the post, but look into SGLang, OP!
In a nutshell, it’s a framework for letting LLMs “fill in blanks” instead of generating entire replies, so you could script in rules as part of the responses as structure for it to grab onto. It’s all locally runnable (with the right hardware, unfortunately).
Also, there are some newer, less sycophantic DM specific models. I can look around if you want.
Is it a virus that affects the brain?
Yes! It’s called engagement optimization. And the worlds collective ignorance “don’t feed the trolls.”
It feels unsustainable, right? Like the value of of this tsunami of advertising has to be inflated, especially with bots/agents taking over traffic. People’s tolerance for junk isn’t infinite. At some point the illusion has to crack, and the advertising bubble will pop and burn the internet/app ecosystems down, hopefully…
This is one of the "smartest" models you can fit on a 24GB GPU now, with no offloading and very little quantization loss. It feels big and insightful, like a better (albeit dry) Llama 3.3 70B with thinking, and with more STEM world knowledge than QwQ 32B, but comfortably fits thanks the new exl3 quantization!
You need to use a backend that support exl3, like (at the moment) text-gen-web-ui or (soon) TabbyAPI.
Tinygrad is (so far) software only, ostensibly sort of a lightweight PyTorch replacement.
Tinygrad is (so far) not really used for much, not even research or tinkering.
Between that and the lead dev's YouTube antics, it kinda seems like hot air to me.