← Directory

koboldcpp

KoboldCpp is a lightweight but powerful AI backend, bundled with KoboldAI Lite frontend.

  • AI
  • Tools & Utilities
KoboldCpp is a lightweight but powerful AI backend, bundled with KoboldAI Lite frontend. https://hub.docker.com/r/koboldai/koboldcpp/

Platform availability

Available on 1 of 9 platforms

  • Unraid
  • TrueNAS (not listed)
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox (not listed)
  • Helm (not listed)
  • Mac (not listed)
  • Windows (not listed)
  • Linux (not listed)

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

20/100 Low

Maintained
17last commit 560 days ago
Popular
176 GitHub stars
Easy to install
351 install platform, install notes, Docker image
Light to run
108 GB minimum RAM

Availability: platform listings confirmed today, links checked 4 days ago

GitHub stars
6
Open issues
0
Last commit
2025-04-01
License
GPL-3.0
Activity
stale

Checked 4 days ago - source: GitHub (oromis995/UnraidKoboldCpp)

Resources & Compatibility

Minimum requirements: 8 GB RAM, 2 CPU core(s), 2 GB disk. ARM (e.g. Raspberry Pi) is not supported.

Checked 14 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Download the Linux binary and make it runnable Download the prebuilt Linux file in your terminal, then run chmod +x on it so the computer can run it; without that permission step, starting KoboldCpp will be blocked. source
  • Load a model before starting the server Start the server with a .gguf model file, or with a folder of model files; without a model file, the server has nothing to answer with and won't be useful. source
  • If you use Nix and an NVIDIA GPU, set CUDA Enable CUDA (the NVIDIA GPU support setting) in your Nix config and list your GPU architecture code, such as sm_75 (a code matching your card); without it, KoboldCpp won't use the GPU and may be slow or fail. source

Fetched 14 days ago - each note links to its source

Alternatives

No alternatives collected yet.

Common questions

Why will KoboldCpp not start on Linux after downloading the binary?

Run chmod +x on the downloaded Linux file so the computer can execute it. Without that permission step, starting KoboldCpp is blocked.

Does KoboldCpp need a model file before the server is started?

Yes. Start the server with a .gguf model file or a folder of model files. Without a model file, the server has nothing to answer with.

Why does KoboldCpp not use an NVIDIA GPU when installed with Nix?

Enable CUDA in your Nix configuration and list your GPU architecture code, such as sm_75. Without that setting, KoboldCpp won't use the GPU and may be slow or fail.

Is a batch size of 1024 a problem in KoboldCpp?

A 1024 batch size is not a problem and is actually recommended for performance if you have the memory.

Does KoboldCpp provide API endpoints for other applications?

Yes. KoboldCpp provides multiple compatible API endpoints for popular web services.

Answers sourced from github.com, github.com, github.com, github.com, www.reddit.com

Community

Collected 13 days ago - public community threads