657
The Rule
(lemmy.ml)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 25 Jul 2024
657 points (100.0% liked)
196
16591 readers
2520 users here now
Be sure to follow the rule before you head out.
Rule: You must post before you leave.
founded 1 year ago
MODERATORS
You're running a 405b param model on 24gb of VRAM, no shit it's not gonna work
--lowvram
Yeah I'm sure that's how they got it to run at all lol, luckily they've fixed a lot of the issues with earlier versions of model runners, I had blue screen running 7b models back then. One of this size might've literally started a fire on my computer
Yeah that's most likely what they did to get it to run at all, but expecting it to produce more than a single token on that hardware is laughable