25
Your best local LLM for low-VRAM (6GB)?
(feddit.org)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 22 May 2026
25 points (90.3% liked)
LocalLLaMA
4738 readers
14 users here now
Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
Rules:
Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.
Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.
Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.
Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.
founded 3 years ago
MODERATORS
Doesn't necessarily need to be very fast, but I don't plan to wait a minute for one simple sentence as well :)
Is that possible without tinkering too much?
I have a Qwen3.6-35b-a3b model running on a dated desktop machine with 4GB VRAM.
I use 8-bit-quant, but also have 48GB normal RAM.
Delivers ~7tk/s, which is already totally usable for most things.
Tried it on my recent Core-i7 company laptop with 8GB VRAM and got 20tk/s.
Oh, and I am also using KoboldCPP (on a Linux foundation).
I'll try my luck and download Qwen3.6-35B-A3B-GGUF. Thanks!
There's been a few videos on Youtube lately discussing using a particular Qwen model that lets you load only particular expert sections at a time onto the GPU and the rest in RAM. This one was the first I watched (https://www.youtube.com/watch?v=8F_5pdcD3HY), I haven't tried it, but it makes sense on why it would work.
with a 20b model on weak hardware you'll be waiting more like 10 minutes. unless the os clobbers your process for using too much memory.