Basalt
Basalt is a fast local inference engine that runs large language models on a single NVIDIA Blackwell GPU, keeping the parts of a model that do not fit in…
Platform availability
Available on 1 of 10 platforms
- Unraid
- TrueNAS (not listed)
- Umbrel (not listed)
- ZimaOS (not listed)
- Proxmox (not listed)
- Helm (not listed)
- Mac (not listed)
- Windows (not listed)
- Linux (not listed)
- Any server (not listed)
Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.
53/100 Fair
- Maintained
- 91last commit 2 days ago
- Popular
- 631,486 GitHub stars
- Easy to install
- 351 install platform, install notes, Docker image
- Light to run
- 304 GB minimum RAM
- Security check
- 05 serious findings, 22 minor, CI, tests
Security check: 5 serious findings
- Uses pillow 12.3.0, which has a known critical vulnerability (PyPI)
- Uses pillow 12.3.0, which has a known critical vulnerability (PyPI)
- Uses pillow 12.3.0, which has a known critical vulnerability (PyPI)
- Uses pillow 12.3.0, which has a known critical vulnerability (PyPI)
- Uses pillow 12.3.0, which has a known critical vulnerability (PyPI)
Automated checks on commit 1771f9d, 19 Nov 2026. Not an audit.
Availability: platform listings confirmed today
- GitHub stars
- 1,486
- Open issues
- 140
- Last commit
- 2026-11-19
- Latest release
- v0.5.0
- License
- MIT
- Activity
- rising
Checked 2 days ago - source: GitHub (jesdga95/basalt)
Resources & Compatibility
Minimum requirements: 4 GB RAM.
Checked 2 days ago - source: Docker Hub image tags and published docs
First-install notes
- Use the 32 GB GPU config before you start Create basalt_config.json with your model file, GPU number, max-context size, and KV cache option (a memory setting for keeping conversation context), then run bin/basalt serve --config basalt_config.json. Missing or wrong values can keep the server from starting, and too much context can use up the GPU memory. source
- Make a Python venv where the binary lives If you are using a release archive, create a .venv folder (a separate Python environment) in the top-level folder and install requirements.txt before starting it. This prevents the server from failing because it cannot find its Python packages. source
- Check where Basalt listens before sharing it By default the server listens on 127.0.0.1 port 8080, which means only the same computer can talk to it. If you need another app or device to use it, change server.host and make port 8080 reachable from where the client is. source
- Blackwell card and recent driver are required The template needs an RTX 50-series (Blackwell) GPU and the NVIDIA Driver plugin on driver 580 or newer; older cards are not supported. Leave room on an SSD pool for the model file and free system RAM for the part of the model that does not fit on the card. source
- Download the model file before first start A model is one .basalt file. Download it into the Models folder from the Unraid terminal, for example: wget -c -P /mnt/cache/appdata/basalt/models https://huggingface.co/jesdga/Qwen3.8-27B-Basalt/resolve/main/Qwen3.8-27B-QUASAR-NVFP4.basalt (-c resumes an interrupted download). source
Fetched 2 days ago - each note links to its source
Variants
Alternatives
No alternatives collected yet.
Common questions
How do I add a model to Basalt?
A model is a single .basalt file. Download it into the Models folder from the terminal.
What client APIs does Basalt expose?
It serves an OpenAI-compatible API and an Anthropic-compatible API, along with a chat page.
Where can model experts be kept when they do not fit in VRAM?
They can be kept in system RAM, and optionally on more GPUs.
What should I preserve when troubleshooting a failed stage?
Preserve the complete command, stdout, stderr, Basalt_log.txt, Basalt_checkpoint.txt, software versions, database configuration, and input checksums.
Answers sourced from basalt-guide.readthedocs.io, github.com
Community
- Community runs: Basalt benchmarks from users' own rigs (forum)
- just bought a mokwheel basalt st. I love this bike does anyone ... - Reddit (reddit)
- How to find Basalt? - Questions - Vintage Story (forum)
Collected 2 days ago - public community threads
