OpenChat-Cuda

A self-hosted, offline, ChatGPT-like chatbot that runs open-source LLMs locally using CUDA on an NVIDIA GPU, keeping conversations private and local.

  • AI
  • Home Automation
  • Productivity
  • Tools & Utilities
A self-hosted, offline, ChatGPT-like chatbot that runs open-source LLMs locally using CUDA on an NVIDIA GPU, keeping conversations private and local.

Platform availability

Available on 1 of 10 platforms

  • Unraid
  • TrueNAS (not listed)
  • Umbrel (not listed)
  • ZimaOS (not listed)
  • Proxmox (not listed)
  • Helm (not listed)
  • Mac (not listed)
  • Windows (not listed)
  • Linux (not listed)
  • Any server (not listed)

Health score?A 0-100 score computed nightly from four factors: maintained (40%: recent commits and steady activity), popular (25%: GitHub stars, log scale), easy to install (20%: platforms, install notes, setup guides, Docker image) and light to run (15%: minimum RAM, ARM support). When the source code has been checked, a fifth factor, security check (20%: automated scan for committed secrets, unsafe desktop-app settings, known-vulnerable dependencies and project hygiene; any serious finding caps it at 20), is added. Factors without data are left out and the rest are rescaled.

17/100 Low

Maintained
0last commit 1170 days ago
Popular
3767 GitHub stars
Easy to install
351 install platform, install notes, Docker image
Light to run
108 GB minimum RAM
Security check
152 serious findings, 16 minor, CI, tests

Availability: platform listings confirmed today, links checked 19 days ago

GitHub stars
67
Open issues
12
Last commit
2023-09-08
Latest release
v1.0.6
License
MIT
Activity
stale

Checked 6 days ago - source: GitHub (edgar971/open-chat)

Resources & Compatibility

Minimum requirements: 8 GB RAM.

Checked 54 days ago - source: Docker Hub image tags and published docs

First-install notes

  • Set your API key first Create a .env.local settings file and add OPENAI_API_KEY (the key that lets the app talk to the AI) plus DEFAULT_MODEL (the model name to use by default). Without a valid key, chats will fail because the app has no permission to use the model. source
  • Point UI to model server If you run a local OpenChat/CUDA backend, set OPENAI_API_HOST (where the app should send chat requests) to its web address and port, and use the key that backend expects. If this is wrong, the UI cannot reach the model and will error when you send a message. source
  • Keep settings file safe Keep .env.local private because it can contain API keys that allow paid or authenticated model access. Anyone who gets it can use that access, so store it only where the app needs it. source
  • NVIDIA GPU required This template only runs on an NVIDIA GPU. Install the NVIDIA driver plugin from the Unraid Community Applications store before starting the container, or the model server will fail to start. source
  • NVIDIA GPU Install the Unraid NVIDIA-DRIVER plugin before starting OpenChat-Cuda. source

Fetched today - each note links to its source

Alternatives

No alternatives collected yet.

Common questions

Which compose command starts the 13B model?

Run docker compose -f docker-compose-13b.yml up -d. The 13B q4_0 model requires at least 16GB RAM.

Which local port should a reverse proxy target?

Use port 3000 because the README says the app is accessible at localhost:3000.

How do I set the model path and context length in Docker?

Add environment variables to the api service, such as MODEL=/path/to/your/model and N_CTX=4096. Additional API settings can also be passed as arguments to run.sh.

Can OpenChat 3.5 7B run on a Pascal GPU?

The source only shows a Reddit question asking whether it works. It does not include a confirmed result.

Answers sourced from github.com, www.reddit.com

Community

Collected 49 days ago - public community threads