NVIDIA NIM Single

NVIDIA NIM (Inference Microservices) is a containerized inference server that runs local large language models on NVIDIA GPUs with CUDA acceleration, exposing…

  • AI
  • Tools & Utilities
NVIDIA NIM (Inference Microservices) is a containerized inference server that runs local large language models on NVIDIA GPUs with CUDA acceleration, exposing an OpenAI-compatible API for developers and homelabbers who want self-hosted LLM access without cloud dependencies.

Platform availability

Available on 1 of 10 platforms

  • Unraid
  • TrueNAS (not listed)
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox (not listed)
  • Helm (not listed)
  • Mac (not listed)
  • Windows (not listed)
  • Linux (not listed)
  • Any server (not listed)

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

48/100 Fair

Maintained
79last commit 30 days ago
Popular
102 GitHub stars
Easy to install
351 install platform, install notes, Docker image
Light to run
No data

Based on 3 of 4 factors.

Availability: platform listings confirmed today, links checked 19 days ago

GitHub stars
2
Open issues
0
Last commit
2026-10-22
License
MIT
Activity
stale

Checked 6 days ago - source: GitHub (PikkonMG/unraid-docker-templates)

Resources & Compatibility

Minimum requirements: 8 CPU core(s).

Checked 54 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Set your API endpoint and key Before sending any call, set invoke_url to your NVIDIA NIM endpoint and add your API key to the Authorization header (the part that carries your key). Without these, the service can't identify you and will reject the request. source
  • Install the Python requests package If you use the Python example, install the requests package first. Without it, the code can't send requests to the API. source
  • Model Selection & VRAM Requirements The default model is meta/llama-3.2-3b-instruct, recommended for GPUs with 12 GB VRAM or less (e.g. RTX 3060, 3070). To switch models, update both the Docker image tag and the NIM_MODEL_NAME variable to matching values. Approximate VRAM usage: Llama 3.2 3B ~6 GB (fits 8-12 GB cards), Mistral 7B ~14 GB (needs 16 GB+), Llama 3.1 8B ~22 GB (needs 24 GB+), Llama 3.1 70B ~80 GB (multi-GPU only). source
  • Install source The Unraid template is available in the PikkonMG Unraid Docker Templates repository. source

Fetched today - each note links to its source

Alternatives

No alternatives collected yet.

Common questions

How do I switch the model in an NVIDIA NIM LLM container?

Update both the Docker image tag and the NIM_MODEL_NAME variable to matching values. The default model is meta/llama-3.2-3b-instruct.

Why does a client call to a NIM endpoint fail authentication?

Set invoke_url to your NIM endpoint and add your API key to the Authorization header. Without these, the service cannot identify you and rejects the request.

How can I reduce memory usage for a NIM running in WSL2?

Set autoMemoryReclaim=dropcache in the WSL2 configuration. These commands require sudo access.

Answers sourced from docs.api.nvidia.com, docs.nvidia.com, docs.nvidia.com

Community

Collected 49 days ago - public community threads