Running LLMs in the Browser Without a Backend

What it does

wllama wraps llama.cpp in a TypeScript layer compiled to WebAssembly, letting you load GGUF models and run inference directly inside the browser. It offers an OpenAI-compatible chat API, pulls models from URLs or Hugging Face, and keeps inference off the main thread so the UI does not freeze. WebGPU acceleration is available, but the library will happily run on CPU via WASM SIMD if no GPU is present.

The interesting bit

Instead of treating the browser as a thin client, wllama treats it as the host. It auto-switches between single-thread and multi-thread WASM builds, splits oversized models into parallel-downloadable chunks to dodge the 2GB ArrayBuffer ceiling, and packs in server-grade features like multimodal and tool-calling support.

Key highlights

• Fully typed, OpenAI-compatible API built in.
• WebGPU layer offloading with automatic fallback to CPU inference.
• Supports multimodal inputs (image and audio) and tool calling in the browser.
• Splits models into chunks for parallel download and to stay under browser memory limits.
• Zero runtime dependencies; runs inside a web worker.

Caveats

• Multi-threading requires Cross-Origin-Embedder-Policy and Cross-Origin-Opener-Policy headers, which can complicate deployment.
• Single files are hard-capped at 2GB by ArrayBuffer limits; larger models must be pre-split.
• IQ (imatrix) quantization is explicitly discouraged for quality and speed reasons in this environment.

Verdict

A strong fit for privacy-centric or offline web demos and client-side assistants where server latency and hosting costs are unwelcome. Teams needing high-throughput batching or guaranteed multi-user concurrency should still reach for a backend stack.