# vLLM

vLLM is an open-source serving engine for large language models (LLMs) that exposes models through an OpenAI-compatible application programming interface (API). It is open source and self-hosted, installable on Docker Hub, Kubernetes (Helm), TrueNAS and Unraid.

Canonical page: https://ourlocalhost.com/app/72/vllm

## Description

vLLM is an open-source serving engine for large language models (LLMs) that exposes models through an OpenAI-compatible application programming interface (API). It uses continuous batching and PagedAttention to provide high-throughput inference with efficient memory use. Self-hosters and developers can use it to serve local models to applications with predictable, production-oriented performance.

## Platform availability

Available on 3 of 9 platforms: Unraid, TrueNAS, Helm.

## Health score

82/100 (Excellent), based on 4 of 4 factors.
- Maintained: 100 (last commit 18 days ago)
- Popular: 99 (93,283 GitHub stars)
- Easy to install: 55 (3 install platforms, install notes, Docker image)
- Light to run: 40 (4 GB minimum RAM, runs on ARM)

## Links

- Docker Hub: https://hub.docker.com/r/vllm/vllm-openai
- Documentation: https://docs.vllm.ai/
- Official website: https://vllm.ai/
- Kubernetes (Helm): https://artifacthub.io/packages/helm/infracloud-charts/vllm
- TrueNAS: https://apps.truenas.com/catalog/vllm/
- Unraid: https://ca.unraid.net/apps/vllm-17eh90i0w140wm?q=vLLM
- GitHub: https://github.com/vllm-project/vllm

## Common questions

### How should I handle model weights before starting vLLM?

Run hf-download.sh to fetch the model before launching vLLM; for private models, set HF_TOKEN first. If the model is missing or the token is wrong, the download fails and there is nothing to serve.

### What cluster setup step is required before starting a multi-node launcher?

Run one-time cluster discovery first with ./autodiscover.sh or ./run-recipe.py --discover to save node settings to .env, then check with --show-env. Without this, the launcher may not know where your nodes are and cluster startup can fail.

### Which port must remain reachable for remote vLLM API clients?

Recipe examples often listen on 0.0.0.0 port 8000, which is the API port clients use to reach vLLM. If you connect from another machine, keep port 8000 open in your firewall or you may not be able to connect.

### Why can two runs of the same prompt produce slightly different logits or log probabilities?

The same requests may be batched differently because of other concurrent requests, changes in batch size, or batch expansion in speculative decoding. These batching variations, combined with Torch numerical instability, can lead to slightly different logits or logprobs.

### Can one OpenAI-compatible vLLM server expose multiple models on the same port?

The OpenAI-compatible server does not currently support serving multiple models on a single port. You can run multiple instances of the server instead.

---
Source: Our Local Host (https://ourlocalhost.com/app/72/vllm), the self-hosted and open-source app directory.