A whole diffusion LLM runtime in Mojo, and it beats llama.cpp on CPU

What it does It runs diffusion language models — LLaDA-8B, LLaDA-1.5, Dream-7B, Dream-Coder, DiffuCoder — locally from quantized GGUF files, on CPU or NVIDIA GPU. The GGUF reader, tokenizer, quantized matmul, GPU kernels, transformer, and denoising loop are all Mojo; the built binary needs no Python at runtime. Diffusion models don’t write left to right — they fill a canvas of masked tokens over fixed steps, committing the ones they’re most sure about — so the runtime problem is a stream of GEMMs, and the project is built around that.

The interesting bit Rather than port a model, the author ported the workload: weights are repacked at load so one AVX-VNNI vpdpbusd handles 8 rows × 4 multiply-adds against int8 activations, and the output projection only computes logits for positions the sampler can actually commit. Every benchmark table is rendered from a result file in bench/results/, and CI fails if a table and its file disagree — reproducibility as a build check.

Key highlights

• Roughly 2× llama.cpp’s step time on CPU for LLaDA-8B Q4_K_M (3,005 vs 5,807 ms/step), with lower peak RSS (5.9 vs 8.4 GB)
• GPU path uses int8 tensor cores for projections and f16 tensor cores with an online-softmax attention; ~18 tokens/s on a modest RTX 3050 laptop GPU
• Correctness is checked against external references: numpy forward passes, llama.cpp logits and tokens, and the gguf package’s dequantization
• Early exit once an end-of-text token follows a fully committed answer — same output, fewer steps
• Hybrid-CPU aware threading: work handed out via an atomic counter so performance cores take more than efficiency cores

Caveats

• Linux on x86-64 with AVX2 and glibc 2.34+ only; the GPU path has been measured on exactly one card (an RTX 3050 Laptop)
• The newer checkpoints (LLaDA-1.5, Dream-Coder, DiffuCoder) have CPU smoke tests and parity results, but not the full benchmark treatment
• The author notes the test laptop’s clocks vary day to day, so absolute milliseconds shouldn’t be compared across dates

Verdict Worth a look if you care about diffusion LLMs, CPU inference, or what Mojo can actually do when someone writes real kernels. If you just want to chat with a model on Windows or a Mac, this isn’t your tool — yet.