← Directory

vLLM

vLLM is an open-source serving engine for large language models (LLMs) that exposes models through an OpenAI-compatible application programming interface (API).

  • AI
vLLM is an open-source serving engine for large language models (LLMs) that exposes models through an OpenAI-compatible application programming interface (API). It uses continuous batching and PagedAttention to provide high-throughput inference with efficient memory use. Self-hosters and developers can use it to serve local models to applications with predictable, production-oriented performance.

Platform availability

Available on 3 of 9 platforms

  • Unraid
  • TrueNAS
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox (not listed)
  • Helm
  • Mac (not listed)
  • Windows (not listed)
  • Linux (not listed)

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

82/100 Excellent

Maintained
100last commit 7 days ago
Popular
9993,283 GitHub stars
Easy to install
553 install platforms, install notes, Docker image
Light to run
404 GB minimum RAM, runs on ARM

Availability: platform listings confirmed today, links checked 16 days ago

GitHub stars
93,283
Open issues
8501
Last commit
2026-10-06
Latest release
v0.31.0
License
Apache-2.0
Activity
rising

Checked 7 days ago - source: GitHub (vllm-project/vllm)

Resources & Compatibility

Minimum requirements: 4 GB RAM, 1 CPU core(s), 10 GB disk. ARM (e.g. Raspberry Pi) is supported.

Checked 15 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Download the model before starting Run the provided hf-download.sh script to fetch the model before launching vLLM; for private models, set HF_TOKEN (your Hugging Face access token) first. If the model is missing or the token is wrong, the download fails and there is nothing to serve. source
  • Run one-time cluster discovery first Run ./autodiscover.sh (or ./run-recipe.py --discover) once to save node settings to .env, then check with --show-env. Without this, the launcher may not know where your nodes are and cluster startup can fail. source
  • Keep API port 8000 reachable Recipe examples often listen on 0.0.0.0 (all network interfaces) port 8000, which is the API port clients use to reach vLLM. If you connect from another machine, keep port 8000 open in your firewall or you may not be able to connect. source

Fetched 15 days ago - each note links to its source

Alternatives

No alternatives collected yet.

Common questions

How should I handle model weights before starting vLLM?

Run hf-download.sh to fetch the model before launching vLLM; for private models, set HF_TOKEN first. If the model is missing or the token is wrong, the download fails and there is nothing to serve.

What cluster setup step is required before starting a multi-node launcher?

Run one-time cluster discovery first with ./autodiscover.sh or ./run-recipe.py --discover to save node settings to .env, then check with --show-env. Without this, the launcher may not know where your nodes are and cluster startup can fail.

Which port must remain reachable for remote vLLM API clients?

Recipe examples often listen on 0.0.0.0 port 8000, which is the API port clients use to reach vLLM. If you connect from another machine, keep port 8000 open in your firewall or you may not be able to connect.

Why can two runs of the same prompt produce slightly different logits or log probabilities?

The same requests may be batched differently because of other concurrent requests, changes in batch size, or batch expansion in speculative decoding. These batching variations, combined with Torch numerical instability, can lead to slightly different logits or logprobs.

Can one OpenAI-compatible vLLM server expose multiple models on the same port?

The OpenAI-compatible server does not currently support serving multiple models on a single port. You can run multiple instances of the server instead.

Answers sourced from docs.vllm.cc, docs.vllm.cc, github.com, github.com, github.com

Community

Collected 15 days ago - public community threads