Self-hosted LLaMA chat for developers who hate API keys

What it does

Serge provides a web chat interface and HTTP API for running LLaMA and Alpaca models locally via llama.cpp. It bundles a SvelteKit frontend and FastAPI backend into a Dockerized stack, handling model serving, chat history, and parameters so you don’t have to wire up the C++ inference engine yourself.

The interesting bit

This is mostly well-crafted glue code: the actual inference is still llama.cpp, and Serge adds the missing web UI, API wrapper, and container plumbing. Its main trick is turning a folder of local model weights into a conversational web app with a Docker-first deployment.

Key highlights

• Fully self-hosted; no cloud API keys or external accounts needed
• FastAPI backend exposes auto-generated API documentation at /api/docs
• Docker-first deployment with volume mounts for model weights and persistent data
• Configurable JWT authentication and session expiry via environment variables
• LangChain sits in the API layer between FastAPI and the llama.cpp Python bindings

Caveats

• Models will crash if the host machine runs out of available RAM
• Windows requires Docker Desktop and WSL2; this is not a lightweight browser-only solution

Verdict

Good for developers who want a private, local chat interface for LLMs without writing frontend code. Skip it if you already have a favorite llama.cpp wrapper or need to run on bare metal without Docker.