Any of you have a self-hosted AI "hub"? (e.g. for LLM, stable-diffusion, ...) (lemmy.ml)

submitted 3 months ago by robber@lemmy.ml to c/selfhosted@lemmy.world

11 comments fedilink hide all child comments

I've been looking into self-hosting LLMs or stable diffusion models using something like LocalAI and / or Ollama and LibreChat.

Some questions to get a nice discussion going:

Any of you have experience with this?
What are your motivations?
What are you using in terms of hardware?
Considerations regarding energy efficiency and associated costs?
What about renting a GPU? Privacy implications?

top 11 comments

sorted by: hot top controversial new old

[-] Audalin@lemmy.world 6 points 3 months ago

Have been using llama.cpp, whisper.cpp, Stable Diffusion for a long while (most often the first one). My "hub" is a collection of bash scripts and a ssh server running.

I typically use LLMs for translation, interactive technical troubleshooting, advice on obscure topics, sometimes coding, sometimes mathematics (though local models are mostly terrible for this), sometimes just talking. Also music generation with ChatMusician.

I use the hardware I already have - a 16GB AMD card (using ROCm) and some DDR5 RAM. ROCm might be tricky to set up for various libraries and inference engines, but then it just works. I don't rent hardware - don't want any data to leave my machine.

My use isn't intensive enough to warrant measuring energy costs.

[-] robber@lemmy.ml 3 points 3 months ago

So you access the models directly via terminal? Is that convenient? Also, do you get satisfying inference speed and quality with a 16GB card?

[-] Audalin@lemmy.world 3 points 3 months ago* (last edited 3 months ago)

Mostly via terminal, yeah. It's convenient when you're used to it - I am.

Let's see, my inference speed now is:

~60-65 tok/s for a 8B model in Q_5_K/Q6_K (entirely in VRAM);
~36 tok/s for a 14B model in Q6_K (entirely in VRAM);
~4.5 tok/s for a 35B model in Q5_K_M (16/41 layers in VRAM);
~12.5 tok/s for a 8x7B model in Q4_K_M (18/33 layers in VRAM);
~4.5 tok/s for a 70B model in Q2_K (44/81 layers in VRAM);
~2.5 tok/s for a 70B model in Q3_K_L (28/81 layers in VRAM).

As of quality, I try to avoid quantisation below Q5 or at least Q4. I also don't see any point in using Q8/f16/f32 - the difference with Q6 is minimal. Other than that, it really depends on the model - for instance, llama-3 8B is smarter than many older 30B+ models.

[-] robber@lemmy.ml 2 points 3 months ago

Thanks! Glad to see the 8x7B performing not too bad - I assume that's a Mistral model? Also, does the CPU significantly affect inference speed in such a setup, do you know?

[-] Audalin@lemmy.world 1 points 3 months ago

If your CPU isn't ancient, it's mostly about memory speed. VRAM is very fast, DDR5 RAM is reasonably fast, swap is slow even on a modern SSD.

8x7B is mixtral, yeah.

[-] rufus@discuss.tchncs.de 6 points 3 months ago* (last edited 3 months ago)

Quite some AI questions coming up in selfhosted in the last few days...

Here's some more communities I'm subscribed to:

And a few inactive ones on lemmy.intai.tech

I'm using koboldcpp and ollama. KoboldCpp is really awesome. In terms of hardware it's an old PC with lots of RAM but no graphics card, so it's quite slow for me. I occasionally rent a cloud GPU instance on runpod.io Not doing anything fancy, mainly role play, recreational stuff and I occasionally ask it to give me creative ideas for something, translate something or re-word or draft a text / email.

[-] Unforeseen@sh.itjust.works 1 points 3 months ago

Thanks for the post, super appreciate the posting of other communties. I think this is a great way to grow Lemmy and create discoverability for niche communities, I'll keep that in mind myself on future opportunities.

[-] snekerpimp@lemmy.world 3 points 3 months ago

Picked up an AMD instinct mi25 to try and do just that. Can get easy-diffusion working after some cussing and voodoo. Cannot get rocm to do ANY llm of any kind, feels like a waste of video ram

Also have a tesla p4 that runs most text-to-image models rather well, but have been unsuccessful at any llm either, even oobabooga can’t seem to run on it.

Have given up because the software stack keeps advancing and leaving my hardware behind. I don’t have $3000 for an a100 or $1300 for an mi100 sooo… until the models can run on older/less powerful hardware, I’m probably sitting out of this game. Even though I’d love to be elbow deep in this one.

[-] robber@lemmy.ml 2 points 3 months ago

Sounds like a rather frustrating journey for you.

[-] snekerpimp@lemmy.world 1 points 3 months ago

It has been. I started in this because I liked picking up kick ass enterprise hardware really cheap and playing around with what it can do. Used enterprise hardware is so damn expensive now, it’s cheaper and easier to do everything with consumer products and use the rx6700 in my gaming rig. Just don’t want that running llms and always on.

[-] fidodo@lemmy.world 3 points 3 months ago* (last edited 3 months ago)

I tried out ollama. It was trivially easy to set up.

Stable diffusion is a bit more work, but any power user should be able to figure it out.

this post was submitted on 24 May 2024

17 points (87.0% liked)

Selfhosted

39206 readers

302 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago

MODERATORS

HybridSarcasm@lemmy.world

HybridSarcasm@lemmy.hybridsarcasm.xyz