# llama.cpp

llama.cpp is a pure C/C++ inference engine for large language models, including Meta's LLaMA. It is open source and self-hosted, installable on ZimaOS and Unraid.

Canonical page: https://ourlocalhost.com/app/2994/llama-cpp

## Description

Llama.cpp is a pure C/C++ inference engine for large language models, including Meta's LLaMA. It runs GGUF models on various hardware, featuring a Vulkan backend for GPU acceleration and an OpenAI-compatible API. It also includes a built-in web user interface for model loading and interaction.

## Platform availability

Available on 2 of 10 platforms: Unraid, ZimaOS.

## Health score

76/100 (Good), based on 4 of 4 factors.
- Maintained: 100 (last commit 7 days ago)
- Popular: 100 (135,045 GitHub stars)
- Easy to install: 45 (2 install platforms, install notes, Docker image)
- Light to run: 10 (16 GB minimum RAM)

## Links

- ZimaOS: https://github.com/IceWhaleTech/CasaOS-AppStore/tree/main/Apps/llama.cpp
- Unraid: https://ca.unraid.net/apps/llama-cpp-0v2s4d30qmhgbv?q=llama.cpp
- Official website: https://llama.app

## Common questions

### How can I load a model directly from Hugging Face?

Use the -hf flag with llama cli or llama serve and pass the model identifier. This downloads and runs the model directly from Hugging Face.

### Why does the API server fail to start?

If the model file path supplied to --model is missing or wrong, the server will not start. Point it at a valid local .gguf file.

### How do I make an NVIDIA GPU do the work?

Install the CUDA wheel that matches your CUDA version and start the server with --n_gpu_layers. Without that flag, it may run mostly on CPU and be slower.

### Does llama serve include a web interface?

Yes, the README lists a built-in web UI that works against llama serve.

### What are quick ways to install it?

You can use the provided install script, download pre-built binaries, run Docker, or build from source. Then launch the OpenAI-compatible API with llama serve.

### Can it run vision-language models?

Yes, llama.cpp is described as supporting LLM and VLM inference. The README shows a VLM session using llama cli.

## Alternatives

- AnythingLLM: AnythingLLM is a local, all-in-one AI app for self-hosters that works with any large language model, providing document chat, retrieval-aug…
- ChatGPT: ChatGPT is a generative artificial intelligence chatbot developed by OpenAI and launched in 2022.
- GPT4ALL: GPT4All runs open-source large language models (LLMs) privately on your own hardware, providing a local chat interface and server that need…
- Jan.ai: Open-source offline AI chat software supporting local LLMs, cloud model connections, custom assistant creation, OpenAI-compatible API, and…
- KoboldCpp: KoboldCpp is a lightweight, self-contained AI backend for text and image generation running GGML and GGUF models on CPU or GPU, bundled wit…
- LM Studio: LM Studio is an application for running and experimenting with local large language models (LLMs) such as gpt-oss, Llama, Gemma, Qwen, and…
- LocalAI: LocalAI is a free, open-source, self-hosted AI engine that provides a drop-in, OpenAI-compatible API for running local models.
- Ollama: Ollama is a local runtime for large language models (LLMs) that provides an application programming interface (API) and command-line interf…
- PocketPal: An app that brings language models directly to your phone.
- lemonade-server: Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs.

---
Source: Our Local Host (https://ourlocalhost.com/app/2994/llama-cpp), the self-hosted and open-source app directory.
