LLM Harbor

LLM Harbor is a local large language model (LLM) management toolkit that unifies llama.cpp, vLLM, and Ollama behind a single CLI on macOS, Windows, and Linux.

  • AI
LLM Harbor is a local large language model (LLM) management toolkit that unifies llama.cpp, vLLM, and Ollama behind a single CLI on macOS, Windows, and Linux. It launches and models-manages containerized backends and exposes them through an API server, so homelab users can serve and switch between local models without juggling separate tools and CLIs.

Platform availability

Available on 4 of 10 platforms

  • Unraid (not listed)
  • TrueNAS (not listed)
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox (not listed)
  • Helm (not listed)
  • Mac
  • Windows
  • Linux
  • Any server

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

70/100 Good

Maintained
100last commit 10 days ago
Popular
703,280 GitHub stars
Easy to install
554 install platforms, install notes
Light to run
108 GB minimum RAM
Security check
70no serious findings, 21 minor, CI, tests

Availability: platform listings confirmed today

GitHub stars
3,280
Open issues
7
Last commit
2026-11-11
Latest release
v0.5.20
License
Apache-2.0
Activity
declining

Checked 3 days ago - source: GitHub (av/harbor)

Resources & Compatibility

Minimum requirements: 8 GB RAM, 10 GB disk.

Checked 3 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Install the terminal tool with cargo If you want the command-line tool, install Rust first and run `cargo install llm`, where cargo is Rust's installer. Without these steps, the `llm` command will not be available in your terminal. source
  • Set your AI provider key first Before using LLM Harbor, set the API key for the AI service you plan to use, such as OPENAI_API_KEY in an environment variable (a named secret value the program reads). Without that key, LLM Harbor cannot call the cloud AI service, so chats and requests fail with authentication errors. source
  • API server uses port 3000 If you run the server example, it listens at 127.0.0.1:3000 with an auth key, meaning your own computer can reach it but other devices usually cannot. Open port 3000 only when another device must use the server, and keep the auth key on because it helps protect your AI setup from uninvited users. source
  • Installing a desktop build For macOS, Windows, and Linux, download the latest native build from the GitHub releases page as an alternative to building from source with cargo. source

Fetched 3 days ago - each note links to its source

Alternatives (8)

  • Agenta - The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.
  • Khoj - Khoj is an open-source AI personal assistant and second brain.
  • Local Deep Research - Local Deep Research is a privacy-focused artificial intelligence (AI) research assistant that performs systematic research on complex quest…
  • LocalAI - LocalAI is a free, open-source, self-hosted AI engine that provides a drop-in, OpenAI-compatible API for running local models.
  • Ollama - Ollama is a local runtime for large language models (LLMs) that provides an application programming interface (API) and command-line interf…
  • ZeroClaw - claw done right 🦀.
  • anythingllm - AnythingLLM is a local, all-in-one AI app for self-hosters that works with any large language model, providing document chat, retrieval-aug…
  • llama.cpp - Llama.cpp is a pure C/C++ inference engine for large language models, including Meta's LLaMA.

Checked 3 days ago - source: AlternativeTo

Common questions

Do I need to build LLM Harbor from source?

No; for macOS, Windows, and Linux, you can download the latest native build from the releases page instead of building from source with cargo.

What credential do I need to set before using the app?

Set the API key for the provider you plan to use, such as OPENAI_API_KEY, in an environment variable. Without it, chats and requests can fail with authentication errors.

Which port should I expose if another device needs the API server?

The server example listens on 127.0.0.1:3000, so other devices usually cannot reach it. Open port 3000 only when another device must use the server, and keep the auth key enabled.

Do I need to write configuration files to wire up the local stack?

It launches a fully pre-wired local LLM stack with backends, frontends, and supporting services without writing any configuration.

Answers sourced from github.com, github.com, www.zendot.org

Community

Collected 3 days ago - public community threads