An 80B model on a 32 GB laptop, in pure C, beating llama.cpp

What it does

Janas-LLM runs large mixture-of-experts models on ordinary hardware, written in C with no dependencies beyond libc. The trick is that weights stay on disk: only the experts each token actually needs get streamed from an NVMe SSD, and frequently-used ones stay cached in RAM. That lets a 48 GB Qwen3-Next-80B run and answer on a 32 GB laptop — 26 tok/s writing, where llama.cpp manages 3.0 on the same machine.

The interesting bit

MoE models are the perfect fit for disk streaming, since each token activates only a few experts — the boring part (I/O discipline, O_DIRECT, expert caching) is exactly where the win comes from. It also splits work across CPU, an integrated Arc GPU via Vulkan, and Intel’s NPU, picking what runs where on its own — and its self-chosen configuration landed within ~2.6% of the best measured setup on every model.

Key highlights

• Faster than llama.cpp on all 13 models compared in Q4_K_M, on the same laptop, files, and day
• Long contexts hold up: at 8,192 tokens it writes 18.2 tok/s vs llama.cpp’s 4.9, with 8-bit KV caches on both
• Small models fly on the NPU: Qwen3.5-0.8B reads at 1,403 tok/s
• Gemma 4 runs whole layers on the GPU; Llama 3 works too, including merged LoRA adapters
• Ships pre-converted models on Hugging Face; .jns files are promised forward-compatible

Caveats

• Explicitly experimental: the API and CLI change without notice, several times a week
• Deliberately hard on the machine — it fills free memory with its expert cache and can push a desktop into swap; one GPU use caused a driver reset
• Developed and tested on exactly one laptop; anything else is unverified territory

Verdict

If you have an NVMe SSD, a modest machine, and a taste for clever systems code, this is worth an evening. If you need stability or run shared hardware, wait until the experiment grows up.