Basalt

Basalt is a fast local inference engine that runs large language models on a single NVIDIA Blackwell GPU, keeping the parts of a model that do not fit in…

  • AI
Basalt is a fast local inference engine that runs large language models on a single NVIDIA Blackwell GPU, keeping the parts of a model that do not fit in graphics memory in system RAM. It supports Qwen3.8-Flash-Next, DeepSeek-V4.1-Flash and the dense Qwen3.8-27B, and serves an OpenAI- and Anthropic-compatible API with a live web UI that shows per-request speed. This template runs an image built from Basalt's own Dockerfile. It is for homelab users with an RTX 50-series card who want fast local model inference without a multi-GPU setup.

Platform availability

Available on 1 of 10 platforms

  • Unraid
  • TrueNAS (not listed)
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox (not listed)
  • Helm (not listed)
  • Mac (not listed)
  • Windows (not listed)
  • Linux (not listed)
  • Any server (not listed)

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

53/100 Fair

Maintained
91last commit 2 days ago
Popular
631,486 GitHub stars
Easy to install
351 install platform, install notes, Docker image
Light to run
304 GB minimum RAM
Security check
05 serious findings, 22 minor, CI, tests

Availability: platform listings confirmed today

GitHub stars
1,486
Open issues
140
Last commit
2026-11-19
Latest release
v0.5.0
License
MIT
Activity
rising

Checked 2 days ago - source: GitHub (jesdga95/basalt)

Resources & Compatibility

Minimum requirements: 4 GB RAM.

Checked 2 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Use the 32 GB GPU config before you start Create basalt_config.json with your model file, GPU number, max-context size, and KV cache option (a memory setting for keeping conversation context), then run bin/basalt serve --config basalt_config.json. Missing or wrong values can keep the server from starting, and too much context can use up the GPU memory. source
  • Make a Python venv where the binary lives If you are using a release archive, create a .venv folder (a separate Python environment) in the top-level folder and install requirements.txt before starting it. This prevents the server from failing because it cannot find its Python packages. source
  • Check where Basalt listens before sharing it By default the server listens on 127.0.0.1 port 8080, which means only the same computer can talk to it. If you need another app or device to use it, change server.host and make port 8080 reachable from where the client is. source
  • Blackwell card and recent driver are required The template needs an RTX 50-series (Blackwell) GPU and the NVIDIA Driver plugin on driver 580 or newer; older cards are not supported. Leave room on an SSD pool for the model file and free system RAM for the part of the model that does not fit on the card. source
  • Download the model file before first start A model is one .basalt file. Download it into the Models folder from the Unraid terminal, for example: wget -c -P /mnt/cache/appdata/basalt/models https://huggingface.co/jesdga/Qwen3.8-27B-Basalt/resolve/main/Qwen3.8-27B-QUASAR-NVFP4.basalt (-c resumes an interrupted download). source

Fetched 2 days ago - each note links to its source

Alternatives

No alternatives collected yet.

Common questions

How do I add a model to Basalt?

A model is a single .basalt file. Download it into the Models folder from the terminal.

What client APIs does Basalt expose?

It serves an OpenAI-compatible API and an Anthropic-compatible API, along with a chat page.

Where can model experts be kept when they do not fit in VRAM?

They can be kept in system RAM, and optionally on more GPUs.

What should I preserve when troubleshooting a failed stage?

Preserve the complete command, stdout, stderr, Basalt_log.txt, Basalt_checkpoint.txt, software versions, database configuration, and input checksums.

Answers sourced from basalt-guide.readthedocs.io, github.com

Community

Collected 2 days ago - public community threads