KoboldCpp
KoboldCpp is a lightweight, self-contained AI backend for text and image generation running GGML and GGUF models on CPU or GPU, bundled with the KoboldAI Lite…
Platform availability
Available on 1 of 10 platforms
- Unraid
- TrueNAS (not listed)
- Umbrel (not listed)
- ZimaOS (not listed)
- Proxmox (not listed)
- Helm (not listed)
- Mac (not listed)
- Windows (not listed)
- Linux (not listed)
- Any server (not listed)
Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.
18/100 Low
- Maintained
- 13last commit 599 days ago
- Popular
- 176 GitHub stars
- Easy to install
- 351 install platform, install notes, Docker image
- Light to run
- 108 GB minimum RAM
Availability: platform listings confirmed today, links checked 43 days ago
- GitHub stars
- 6
- Open issues
- 0
- Last commit
- 2025-04-01
- License
- GPL-3.0
- Activity
- stale
Checked 19 days ago - source: GitHub (oromis995/UnraidKoboldCpp)
Resources & Compatibility
Minimum requirements: 8 GB RAM, 2 CPU core(s), 2 GB disk. ARM (e.g. Raspberry Pi) is not supported.
Checked 54 days ago - source: Docker Hub image tags and published docs
First-install notes
- Download the Linux binary and make it runnable Download the prebuilt Linux file in your terminal, then run chmod +x on it so the computer can run it; without that permission step, starting KoboldCpp will be blocked. source
- Load a model before starting the server Start the server with a .gguf model file, or with a folder of model files; without a model file, the server has nothing to answer with and won't be useful. source
- If you use Nix and an NVIDIA GPU, set CUDA Enable CUDA (the NVIDIA GPU support setting) in your Nix config and list your GPU architecture code, such as sm_75 (a code matching your card); without it, KoboldCpp won't use the GPU and may be slow or fail. source
Fetched 54 days ago - each note links to its source
Variants
Alternatives
No alternatives collected yet.
Common questions
Why will KoboldCpp not start on Linux after downloading the binary?
Run chmod +x on the downloaded Linux file so the computer can execute it. Without that permission step, starting KoboldCpp is blocked.
Does KoboldCpp need a model file before the server is started?
Yes. Start the server with a .gguf model file or a folder of model files. Without a model file, the server has nothing to answer with.
Why does KoboldCpp not use an NVIDIA GPU when installed with Nix?
Enable CUDA in your Nix configuration and list your GPU architecture code, such as sm_75. Without that setting, KoboldCpp won't use the GPU and may be slow or fail.
Is a batch size of 1024 a problem in KoboldCpp?
A 1024 batch size is not a problem and is actually recommended for performance if you have the memory.
Does KoboldCpp provide API endpoints for other applications?
Yes. KoboldCpp provides multiple compatible API endpoints for popular web services.
Answers sourced from github.com, github.com, github.com, github.com, www.reddit.com
Community
- KoboldCpp - Combining all the various ggml.cpp CPU LLM ... - Reddit (reddit)
- So, kobold.cpp took over nowadays? : r/KoboldAI - Reddit (reddit)
- The KoboldCpp FAQ and Knowledgebase - Reddit (reddit)
- OpenKlyde - A Self Hosted AI Discord Bot : r/KoboldAI - Reddit (reddit)
- The KoboldCpp FAQ and Knowledgebase - Reddit (reddit)
- PSA: This koboldcpp fork by "kalomaze" has amazing CPU ... - Reddit (reddit)
Collected 53 days ago - public community threads