LLM inference that never leaves the browser tab

What it does WebLLM is an inference engine that downloads and executes quantized LLMs—including Llama, Phi, Mistral, Qwen, and others—entirely within the browser using WebGPU acceleration. It exposes a local MLCEngine that mimics the OpenAI chat-completions interface, so existing client code can point to an offline model without rewriting requests. No server round-trips are required after the initial model fetch.

The interesting bit Instead of treating the browser as a thin client, WebLLM compiles models to an MLC format and pushes heavy computation into WebWorkers or ServiceWorkers, keeping the main thread free while still leveraging GPU acceleration.

Key highlights

• Runs fully offline after the first model download, with no backend infrastructure.
• OpenAI API compatibility for streaming, JSON-mode structured generation, and logit control.
• Supports multiple cache backends: Cache API, IndexedDB, OPFS, and an experimental Cross-Origin Storage API.
• Broad model support including Llama 3, Phi 3, Gemma, Mistral, and Qwen.
• Offload inference to Web Workers or Service Workers to protect UI responsiveness.

Caveats

• Function calling is marked work-in-progress.
• First-time model downloads can be slow; the README warns you must handle the asynchronous load gracefully.
• The experimental Cross-Origin Storage backend requires a Chrome extension and lacks programmatic cache clearing.

Verdict Front-end developers who want private, client-side AI or need to drop server inference costs should look here. If you need multi-user concurrency or massive context windows beyond consumer GPU limits, this is the wrong stack.