llama.cpp
Llama.cpp is a pure C/C++ inference engine for large language models, including Meta's LLaMA.
Platform availability
Available on 2 of 10 platforms
Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.
76/100 Good
- Maintained
- 100last commit 7 days ago
- Popular
- 100135,045 GitHub stars
- Easy to install
- 452 install platforms, install notes, Docker image
- Light to run
- 1016 GB minimum RAM
Availability: platform listings confirmed today, links checked 3 days ago
- GitHub stars
- 135,045
- Open issues
- 2622
- Last commit
- 2026-11-14
- Latest release
- v1.0.0
- License
- MIT
- Activity
- rising
Checked 7 days ago - source: GitHub (ggml-org/llama.cpp)
Resources & Compatibility
Minimum requirements: 16 GB RAM.
Checked 54 days ago - source: Docker Hub image tags and published docs
First-install notes
- Start with the easy CPU wheel If you're just trying it, install the pre-built CPU wheel using the docs' pip command with the CPU extra-index-url. This avoids building the code from scratch and gives you a working baseline first. source
- Install and start the server with a model Install the server extras with pip, then start the server by pointing the --model option at your local .gguf model file. If the model file is missing or the path is wrong, the server will not start. source
- NVIDIA GPU: use the matching CUDA wheel For an NVIDIA GPU, install the CUDA wheel that matches your CUDA version, such as the cu121 example in the docs. Then use --n_gpu_layers when starting the server so some of the work runs on the GPU; without that, it may run mostly on CPU and be slower. source
Fetched 53 days ago - each note links to its source
Variants
Alternatives (10)
- AnythingLLM - AnythingLLM is a local, all-in-one AI app for self-hosters that works with any large language model, providing document chat, retrieval-aug…
- ChatGPT - ChatGPT is a generative artificial intelligence chatbot developed by OpenAI and launched in 2022.
- GPT4ALL - GPT4All runs open-source large language models (LLMs) privately on your own hardware, providing a local chat interface and server that need…
- Jan.ai - Open-source offline AI chat software supporting local LLMs, cloud model connections, custom assistant creation, OpenAI-compatible API, and…
- KoboldCpp - KoboldCpp is a lightweight, self-contained AI backend for text and image generation running GGML and GGUF models on CPU or GPU, bundled wit…
- LM Studio - LM Studio is an application for running and experimenting with local large language models (LLMs) such as gpt-oss, Llama, Gemma, Qwen, and…
- LocalAI - LocalAI is a free, open-source, self-hosted AI engine that provides a drop-in, OpenAI-compatible API for running local models.
- Ollama - Ollama is a local runtime for large language models (LLMs) that provides an application programming interface (API) and command-line interf…
- PocketPal - An app that brings language models directly to your phone.
- lemonade-server - Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs.
Checked 54 days ago - source: AlternativeTo
Common questions
How can I load a model directly from Hugging Face?
Use the -hf flag with llama cli or llama serve and pass the model identifier. This downloads and runs the model directly from Hugging Face.
Why does the API server fail to start?
If the model file path supplied to --model is missing or wrong, the server will not start. Point it at a valid local .gguf file.
How do I make an NVIDIA GPU do the work?
Install the CUDA wheel that matches your CUDA version and start the server with --n_gpu_layers. Without that flag, it may run mostly on CPU and be slower.
Does llama serve include a web interface?
Yes, the README lists a built-in web UI that works against llama serve.
What are quick ways to install it?
You can use the provided install script, download pre-built binaries, run Docker, or build from source. Then launch the OpenAI-compatible API with llama serve.
Can it run vision-language models?
Yes, llama.cpp is described as supporting LLM and VLM inference. The README shows a VLM session using llama cli.
Answers sourced from github.com, llama-cpp-python.readthedocs.io
Community
- Short guide to hosting your own llama.cpp openAI compatible ... - Reddit (reddit)
- llamacpp - Reddit (reddit)
- Anyone is using llamacpp for real? : r/LocalLLaMA - Reddit (reddit)
- Runtime Local LLM (llama.cpp) - Epic Developer Community Forums (forum)
Collected 49 days ago - public community threads