LLMKube

LLMKube is a Kubernetes operator that runs self-hosted large language models (LLMs) on local hardware.

  • AI
  • Infrastructure
LLMKube is a Kubernetes operator that runs self-hosted large language models (LLMs) on local hardware. It exposes OpenAI-compatible endpoints for NVIDIA, AMD, Intel, and Apple Silicon hosts from a single configuration spec. It suits platform teams that need consistent model deployment across mixed or air-gapped GPU fleets.

Platform availability

Available on 4 of 10 platforms

  • Unraid (not listed)
  • TrueNAS (not listed)
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox (not listed)
  • Helm (not listed)
  • Mac
  • Windows
  • Linux
  • Any server

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

75/100 Good

Maintained
100last commit 3 days ago
Popular
49267 GitHub stars
Easy to install
554 install platforms, install notes
Light to run
No data
Security check
76no serious findings, 4 minor, CI, tests

Based on 4 of 5 factors.

Availability: platform listings confirmed today

GitHub stars
267
Open issues
139
Last commit
2026-11-18
Latest release
v0.10.19
License
Apache-2.0
Activity
rising

Checked 3 days ago - source: GitHub (defilantech/LLMKube)

Resources & Compatibility

Requirements not published yet.

Checked 3 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Install cluster tools first Install Minikube (the small local Kubernetes cluster that LLMKube can run in), kubectl (the command tool for the cluster), and Helm (the cluster app installer) using the commands for your OS. Without these, you cannot start the cluster or install LLMKube. source
  • Start Minikube with enough space Start Minikube with the recommended 4 CPUs, 6GB memory, and 20GB disk, then run minikube status until the cluster says ready. Too little memory or disk can make model containers crash or prevent LLMKube from running workloads reliably. source
  • Install LLMKube and check the controller Add the LLMKube Helm repository and run helm install llmkube llmkube/llmkube --namespace llmkube-system --create-namespace, then use kubectl get pods -n llmkube-system to confirm the controller pod is Running. If this manager is not Running, your Model and InferenceService objects will not be processed, so LLMKube will not start the models. source
  • Apple Silicon On Apple Silicon, LLMKube runs inference as native macOS processes instead of containers. source

Fetched 3 days ago - each note links to its source

Alternatives (4)

  • KubeAI
  • NVIDIA NIM - NVIDIA NIM is a set of accelerated inference microservices that allow organizations to run AI models on NVIDIA GPUs anywhere.
  • Ollama - Ollama is a local runtime for large language models (LLMs) that provides an application programming interface (API) and command-line interf…
  • llm-d

Checked 3 days ago - source: AlternativeTo

Common questions

How do I add an Apple Silicon Mac to LLMKube without exposing cluster credentials?

Use the out-of-cluster Metal Agent. Create a one-use join token from the cluster with llmkube metal token create, paste the printed join string on the Mac with llmkube metal join, and deploy models with accelerator: metal. The join string expires in 30 minutes and the Mac does not receive cluster credentials.

What Minikube memory and disk size should I use before installing LLMKube?

Start Minikube with about 4 CPUs, 6GB of memory, and 20GB of disk, then wait for minikube status to say ready. Too little memory or disk can crash model containers or stop LLMKube from running workloads reliably.

Where can LLMKube download models from, and are they stored locally?

LLMKube can download model files from Hugging Face, S3, OCI, or a URL. It caches them, so the same model is available locally after the initial download.

For a Metal-backed inference service, where should a reverse proxy point?

Point it at the ordinary Kubernetes Service created by LLMKube for that InferenceService. The documented example calls that service on port 8080 at the OpenAI-compatible chat completions endpoint.

Can I rely on the LLMKube CRD fields remaining stable?

LLMKube is pre-1.0, and every CRD is v1alpha1. Fields can change between minor releases.

How are off-cluster agents kept updated?

LLMKube uses declarative, health-gated self-update for off-cluster agents. The edge paths mentioned are helm and brew.

Answers sourced from dev.to, github.com, github.com, llmkube.com

Community

Collected 3 days ago - public community threads