llama.cpp

Llama.cpp is a pure C/C++ inference engine for large language models, including Meta's LLaMA.

  • AI
  • Other
  • Productivity
  • Tools & Utilities
Llama.cpp is a pure C/C++ inference engine for large language models, including Meta's LLaMA. It runs GGUF models on various hardware, featuring a Vulkan backend for GPU acceleration and an OpenAI-compatible API. It also includes a built-in web user interface for model loading and interaction.

Platform availability

Available on 2 of 10 platforms

  • Unraid
  • TrueNAS (not listed)
  • Umbrel (not listed)
  • ZimaOS
  • Proxmox (not listed)
  • Helm (not listed)
  • Mac (not listed)
  • Windows (not listed)
  • Linux (not listed)
  • Any server (not listed)

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

76/100 Good

Maintained
100last commit 7 days ago
Popular
100135,045 GitHub stars
Easy to install
452 install platforms, install notes, Docker image
Light to run
1016 GB minimum RAM

Availability: platform listings confirmed today, links checked 3 days ago

GitHub stars
135,045
Open issues
2622
Last commit
2026-11-14
Latest release
v1.0.0
License
MIT
Activity
rising

Checked 7 days ago - source: GitHub (ggml-org/llama.cpp)

Resources & Compatibility

Minimum requirements: 16 GB RAM.

Checked 54 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Start with the easy CPU wheel If you're just trying it, install the pre-built CPU wheel using the docs' pip command with the CPU extra-index-url. This avoids building the code from scratch and gives you a working baseline first. source
  • Install and start the server with a model Install the server extras with pip, then start the server by pointing the --model option at your local .gguf model file. If the model file is missing or the path is wrong, the server will not start. source
  • NVIDIA GPU: use the matching CUDA wheel For an NVIDIA GPU, install the CUDA wheel that matches your CUDA version, such as the cu121 example in the docs. Then use --n_gpu_layers when starting the server so some of the work runs on the GPU; without that, it may run mostly on CPU and be slower. source

Fetched 53 days ago - each note links to its source

Alternatives (10)

  • AnythingLLM - AnythingLLM is a local, all-in-one AI app for self-hosters that works with any large language model, providing document chat, retrieval-aug…
  • ChatGPT - ChatGPT is a generative artificial intelligence chatbot developed by OpenAI and launched in 2022.
  • GPT4ALL - GPT4All runs open-source large language models (LLMs) privately on your own hardware, providing a local chat interface and server that need…
  • Jan.ai - Open-source offline AI chat software supporting local LLMs, cloud model connections, custom assistant creation, OpenAI-compatible API, and…
  • KoboldCpp - KoboldCpp is a lightweight, self-contained AI backend for text and image generation running GGML and GGUF models on CPU or GPU, bundled wit…
  • LM Studio - LM Studio is an application for running and experimenting with local large language models (LLMs) such as gpt-oss, Llama, Gemma, Qwen, and…
  • LocalAI - LocalAI is a free, open-source, self-hosted AI engine that provides a drop-in, OpenAI-compatible API for running local models.
  • Ollama - Ollama is a local runtime for large language models (LLMs) that provides an application programming interface (API) and command-line interf…
  • PocketPal - An app that brings language models directly to your phone.
  • lemonade-server - Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs.

Checked 54 days ago - source: AlternativeTo

Common questions

How can I load a model directly from Hugging Face?

Use the -hf flag with llama cli or llama serve and pass the model identifier. This downloads and runs the model directly from Hugging Face.

Why does the API server fail to start?

If the model file path supplied to --model is missing or wrong, the server will not start. Point it at a valid local .gguf file.

How do I make an NVIDIA GPU do the work?

Install the CUDA wheel that matches your CUDA version and start the server with --n_gpu_layers. Without that flag, it may run mostly on CPU and be slower.

Does llama serve include a web interface?

Yes, the README lists a built-in web UI that works against llama serve.

What are quick ways to install it?

You can use the provided install script, download pre-built binaries, run Docker, or build from source. Then launch the OpenAI-compatible API with llama serve.

Can it run vision-language models?

Yes, llama.cpp is described as supporting LLM and VLM inference. The README shows a VLM session using llama cli.

Answers sourced from github.com, llama-cpp-python.readthedocs.io

Community

Collected 49 days ago - public community threads