145B parameters, one consumer GPU, no cloud in sight

What it does Edge0 is an open-source inference framework for large mixture-of-experts models on consumer devices. The current release runs 145B and 120B latent-MoE models (derived from Qwen3.8-Flash-Next, trained natively at 4 bits) on a single NVIDIA GPU under Linux or Windows, via a unified C++ engine, edge0-infer, with CLI, an OpenAI-compatible local server, and chat apps. The earlier 35B/8B release targeted phones and laptops — iOS, Android, macOS, Windows — using a GGML-based native engine that streams weights from SSD.

The interesting bit The trick is treating storage bandwidth as a first-class citizen: a router trained to predict expert loads a step ahead lets the engine prefetch the right expert weights from SSD before they’re needed. Their arXiv report calls this “the other half of the memory wall” — most MoE work optimizes compute, this optimizes the wait for weights to arrive.

Key highlights

• 145B Q3_K on an RTX 5090 D 24 GB: 121.2 tok/s agent decode, 3554.9 tok/s on 16K prefill.
• Runs on modest setups too — a 16 GiB VRAM + 20 GiB RAM machine still hits ~45 tok/s agent decode.
• Quality holds up: 145B posts the best DeepSWE 1.1 (64.6) and Terminal-Bench 2.1 (86.5) scores in their six-model comparison.
• One C++ API across macOS, iOS, Android, and Windows for the 35B tier; Apache 2.0 licensed.
• 512 experts per layer, K=10, with quantized latent experts (1792 for 145B).

Caveats

• The 145B/120B models need an NVIDIA GPU with CUDA 13 — no Mac or phone support for the current flagship tier.
• The 8B checkpoint exists but native-engine support is still “planned.”
• The project has churned engines already: the original Python framework and per-platform engines were replaced and archived at pre-edge0-infer.

Verdict Worth a look if you want frontier-ish model quality without renting a datacenter — especially the SSD-streaming router idea. Skip it if you’re on Apple Silicon hoping to run the 145B; that combination doesn’t exist yet.