vLLM

vLLM is an open-source serving engine for large language models (LLMs) that exposes models through an OpenAI-compatible application programming interface (API).

  • AI
vLLM is an open-source serving engine for large language models (LLMs) that exposes models through an OpenAI-compatible application programming interface (API). It uses continuous batching and PagedAttention to provide high-throughput inference with efficient memory use. Self-hosters and developers can use it to serve local models to applications with predictable, production-oriented performance.

Platform availability

Available on 4 of 10 platforms

  • Unraid
  • TrueNAS
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox
  • Helm
  • Mac (not listed)
  • Windows (not listed)
  • Linux (not listed)
  • Any server (not listed)

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

84/100 Excellent

Maintained
100last commit 23 days ago
Popular
10095,856 GitHub stars
Easy to install
654 install platforms, install notes, Docker image
Light to run
404 GB minimum RAM, runs on ARM

Availability: platform listings confirmed today, links checked 10 days ago

GitHub stars
95,856
Open issues
9583
Last commit
2026-10-29
Latest release
v0.32.0
License
Apache-2.0
Activity
rising

Checked 23 days ago - source: GitHub (vllm-project/vllm)

Resources & Compatibility

Minimum requirements: 4 GB RAM, 1 CPU core(s), 10 GB disk. ARM (e.g. Raspberry Pi) is supported.

Checked 54 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Download the model before starting Run the provided hf-download.sh script to fetch the model before launching vLLM; for private models, set HF_TOKEN (your Hugging Face access token) first. If the model is missing or the token is wrong, the download fails and there is nothing to serve. source
  • Run one-time cluster discovery first Run ./autodiscover.sh (or ./run-recipe.py --discover) once to save node settings to .env, then check with --show-env. Without this, the launcher may not know where your nodes are and cluster startup can fail. source
  • Keep API port 8000 reachable Recipe examples often listen on 0.0.0.0 (all network interfaces) port 8000, which is the API port clients use to reach vLLM. If you connect from another machine, keep port 8000 open in your firewall or you may not be able to connect. source

Fetched 54 days ago - each note links to its source

Alternatives

No alternatives collected yet.

Common questions

How should I handle model weights before starting vLLM?

Run hf-download.sh to fetch the model before launching vLLM; for private models, set HF_TOKEN first. If the model is missing or the token is wrong, the download fails and there is nothing to serve.

What cluster setup step is required before starting a multi-node launcher?

Run one-time cluster discovery first with ./autodiscover.sh or ./run-recipe.py --discover to save node settings to .env, then check with --show-env. Without this, the launcher may not know where your nodes are and cluster startup can fail.

Which port must remain reachable for remote vLLM API clients?

Recipe examples often listen on 0.0.0.0 port 8000, which is the API port clients use to reach vLLM. If you connect from another machine, keep port 8000 open in your firewall or you may not be able to connect.

Why can two runs of the same prompt produce slightly different logits or log probabilities?

The same requests may be batched differently because of other concurrent requests, changes in batch size, or batch expansion in speculative decoding. These batching variations, combined with Torch numerical instability, can lead to slightly different logits or logprobs.

Can one OpenAI-compatible vLLM server expose multiple models on the same port?

The OpenAI-compatible server does not currently support serving multiple models on a single port. You can run multiple instances of the server instead.

Answers sourced from docs.vllm.cc, docs.vllm.cc, github.com, github.com, github.com

Community

Collected 54 days ago - public community threads