13
Running Local LLMs with Ollama on openSUSE Tumbleweed
(news.opensuse.org)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 14 Jul 2025
13 points (93.3% liked)
LocalLLaMA
2978 readers
29 users here now
Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
Rules:
Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.
Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.
Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.
Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.
founded 2 years ago
MODERATORS
Thanks for the random suggestion! Installed it already. Sadly as a drop-in replacement it doesn't provide any speedup on my old machine, it's exactly the same number of tokens per second... Guess I have to learn about the ik_llama.cpp and pick a different quantization of my favourite model.
What model size/family? What GPU? What context length? There are many different backends with different strengths, but I can tell you the optimal way to run it and the quantization you should run with a bit more specificity, heh.
CPU-only. It's an old Xeon workstation without any GPU, since I mostly do one-off AI tasks at home and I never felt any urge to buy one (yet). Model size woul be something between 7B and 32B with that. Context length is something like 8128 tokens. I have a bit less than 30GB of RAM to waste since I'm doing other stuff on that machine as well.
And I'm picky with the models. I dislike the condescending tone of ChatGPT and newer open-weight models. I don't want it to blabber or praise me for my "genious" ideas. It should be creative, have some storywriting abilities, be uncensored and not overly agreeable. Best model I found for that is Mistral-Nemo-Instruct. And I currently run a Q4_K_M quant of it. That does about 2.5 t/s on my computer (which isn't a lot, but somewhat acceptable for what I do). Mistral-Nemo isn't the latest and greatest any more. But I really prefer it's tone of speaking and it performs well on a wide variety of tasks. And I mostly do weird things with it. Let it give me creative advice, be a dungeon master or an late 80s text adventure. Or mimick a radio moderator and feed it into TTS for a radio show. Or write a book chapter or a bad rap song. I'm less concerned with the popular AI use-cases like answer factual questions or write computer code. So I'd like to switch to a newer, more "intelligent" model. But that proves harder than I imagined.
(Occasionally I do other stuff as well, but that's a far and in-between. So I'll rent a datacenter GPU on runpod.io for a few bucks an hour. That's the main reason why I didn't buy an own GPU yet.)
One more thing, you don't have to get something shiny and new to speed LLMs up. Even if you have like a 4-6GB GPU collecting dust somehwere, you can still use it to partially offload MoE models to great effect.
Try any of Latitude's series. They're 'uninhibited' dungeonmaster models, but they should be smart enough (and retain enough of that personality) for some flexibility:
https://huggingface.co/LatitudeGames
Perhaps more optimally for your hardware, try this:
https://huggingface.co/Gryphe/Pantheon-Proto-RP-1.8-30B-A3B
It's trained from Qwen A3B base, not instruct. Base models usually don't have the severe ChatGPT-isms you describe, hence while I haven't personally tried this model, it seems promising. And it should be fast on your Xeon.