cross-posted from: https://lemmy.world/post/440237
The latest update of Koboldcpp v1.32 brings significant performance boosts to AI computations at home, enabling faster generation speeds and improved memory management for several AI models like MPT, GPT-2, GPT-J and GPT-NeoX, plus upgraded K-Quant matmul kernels for OpenCL.
By leveraging GPU power via OpenCL and implementing optimized programming techniques, it allows hobbyist and enthusiast users to run these advanced models more efficiently on home hardware.
LostRuin's Koboldcpp v1.32 GitHub Patch Notes
- Ported the optimized K-Quant CUDA kernels to OpenCL ! This speeds up K-Quants generation speed by about 15% with CL (Special thanks: @0cc4m)
- Implemented basic GPU offloading for MPT, GPT-2, GPT-J and GPT-NeoX via OpenCL! It still keeps a copy of the weights in RAM, but generation speed for these models should now be much faster! (50% speedup for GPT-J, and even WizardCoder is now 30% faster for me.)
- Implemented scratch buffers for the latest versions of all non-llama architectures except RWKV (MPT, GPT-2, NeoX, GPT-J), BLAS memory usage should be much lower on average, and larger BLAS batch sizes will be usable on these models.
- Merged GPT-Tokenizer improvements for non-llama models. Support Starcoder special added tokens. Coherence for non-llama models should be improved.
- Updated Lite, pulled updates from upstream, various minor bugfixes.
To use, download and run the koboldcpp.exe, which is a one-file pyinstaller. Alternatively, drag and drop a compatible ggml model on top of the .exe, or run it and manually select the model in the popup dialog.
Once loaded, you can connect like this (or use the full koboldai client):
http://localhost:5001
For more information, be sure to run the program with the --help flag.
Want to run AI models at home? Checkout Koboldcpp on Github, an inference engine made by LostRuins, or any of the other many options you can download at home for free.
