Local LLMs in a Desktop Skin, Plus an OpenAI-Compatible API

What it does

Jan is a Tauri-based desktop application that downloads and runs open-source LLMs—Llama, Gemma, Qwen, and others—directly on your hardware via llama.cpp. It also connects to commercial APIs like OpenAI, Anthropic, and Groq, letting you switch between offline models and cloud providers inside the same interface. A built-in OpenAI-compatible server at localhost:1337 lets other applications piggyback on your local models.

The interesting bit

Rather than forcing a binary choice between local and cloud, Jan treats them as interchangeable backends under one roof. Model Context Protocol support nudges it toward agentic workflows, so your chat client can double as a lightweight command center for automated tasks.

Key highlights

• Runs LLMs fully offline on Windows, macOS, and Linux with GPU acceleration where available.
• Ships with an OpenAI-compatible local API server so other tools can route requests through your own hardware.
• Toggles between local models and cloud providers (OpenAI, Anthropic, Mistral, Groq, MiniMax) without leaving the app.
• Supports custom assistants and MCP integration for task-specific agentic behavior.
• Apache 2.0 licensed.

Caveats

• Local model requirements scale fast: the README suggests 8 GB RAM for a 3B model, 16 GB for 7B, and 32 GB for 13B.
• Linux ARM64 builds exist, but getting them requires following a GitHub issue comment rather than a standard release download.

Verdict

Developers who want a polished, privacy-first chat interface with a built-in local API server will find Jan a practical daily driver. If you only ever use cloud models and never touch local hardware, you are not the target audience.