NVIDIA NIM Single
NVIDIA NIM (Inference Microservices) is a containerized inference server that runs local large language models on NVIDIA GPUs with CUDA acceleration, exposing…
Platform availability
Available on 1 of 10 platforms
- Unraid
- TrueNAS (not listed)
- Umbrel (not listed)
- ZimaOS (not listed)
- Proxmox (not listed)
- Helm (not listed)
- Mac (not listed)
- Windows (not listed)
- Linux (not listed)
- Any server (not listed)
Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.
48/100 Fair
- Maintained
- 79last commit 30 days ago
- Popular
- 102 GitHub stars
- Easy to install
- 351 install platform, install notes, Docker image
- Light to run
- No data
Based on 3 of 4 factors.
Availability: platform listings confirmed today, links checked 19 days ago
- GitHub stars
- 2
- Open issues
- 0
- Last commit
- 2026-10-22
- License
- MIT
- Activity
- stale
Checked 6 days ago - source: GitHub (PikkonMG/unraid-docker-templates)
Resources & Compatibility
Minimum requirements: 8 CPU core(s).
Checked 54 days ago - source: Docker Hub image tags and published docs
First-install notes
- Set your API endpoint and key Before sending any call, set invoke_url to your NVIDIA NIM endpoint and add your API key to the Authorization header (the part that carries your key). Without these, the service can't identify you and will reject the request. source
- Install the Python requests package If you use the Python example, install the requests package first. Without it, the code can't send requests to the API. source
- Model Selection & VRAM Requirements The default model is meta/llama-3.2-3b-instruct, recommended for GPUs with 12 GB VRAM or less (e.g. RTX 3060, 3070). To switch models, update both the Docker image tag and the NIM_MODEL_NAME variable to matching values. Approximate VRAM usage: Llama 3.2 3B ~6 GB (fits 8-12 GB cards), Mistral 7B ~14 GB (needs 16 GB+), Llama 3.1 8B ~22 GB (needs 24 GB+), Llama 3.1 70B ~80 GB (multi-GPU only). source
- Install source The Unraid template is available in the PikkonMG Unraid Docker Templates repository. source
Fetched today - each note links to its source
Variants
Alternatives
No alternatives collected yet.
Common questions
How do I switch the model in an NVIDIA NIM LLM container?
Update both the Docker image tag and the NIM_MODEL_NAME variable to matching values. The default model is meta/llama-3.2-3b-instruct.
Why does a client call to a NIM endpoint fail authentication?
Set invoke_url to your NIM endpoint and add your API key to the Authorization header. Without these, the service cannot identify you and rejects the request.
How can I reduce memory usage for a NIM running in WSL2?
Set autoMemoryReclaim=dropcache in the WSL2 configuration. These commands require sudo access.
Answers sourced from docs.api.nvidia.com, docs.nvidia.com, docs.nvidia.com
Community
- Demystifying NVIDIA NIM: Empowering AI Deployment and Monitoring - Reddit (reddit)
- Anyone using Nim for AI? : r/nim - Reddit (reddit)
- Latest NVIDIA NIM topics - NVIDIA Developer Forums (forum)
- Can self-hosted (docker) NIMs function in an environment without Nvidia ... (forum)
- NVIDIA NIM FAQ - NVIDIA NIM - NVIDIA Developer Forums (forum)
- Nim forum (forum)
Collected 49 days ago - public community threads