A C engine that treats your SSD as swap for a 2.78T-parameter model

What it does

WASTE is an embeddable C inference engine with no third-party runtime dependencies. It keeps the shared trunk of a mixture-of-experts model in RAM and streams only the activated experts directly from NVMe storage, using whatever memory remains as a bounded cache. The result is that the full 2.78-trillion-parameter Kimi K3 model runs on a 64 GB MacBook Pro at roughly 0.6 tokens per second from a 982 GB on-disk container.

The interesting bit

The project is deliberately narrow: it treats RAM as a cache for the disk, not the other way around. A lookahead router prefetches experts before the real router confirms they are needed, and the code itself is written by LLMs—because, at this scale, the authors argue that is the only way to iterate fast enough on new algorithms.

Key highlights

• The full, undistilled Kimi K3 (2.78 T params, 982 GB container) decodes at 0.45–0.62 tok/s on a 64 GB MacBook Pro with an internal SSD.
• Experts are quantized to 3-bit residual vector quantization, while shared weights stay at 4 or 8 bits; final logits agree with PyTorch within 3.6e-06.
• Linear attention and a compressed latent KV cache shrink the 4K-context cache to about 0.21 GB instead of 11.25 GB.
• The engine needs only libc and pthreads for inference; no BLAS, CUDA, or Python runtime is required.
• A smaller Kimi-Linear 48B model fits in a 19 GB container, needs 1.28 GB of RAM, and runs at about 10.7 tok/s on the same hardware.

Caveats

• Giving the process more memory is not always faster: on the test machine, expanding the expert cache beyond 17 GB caused throughput to drop eightfold because cache hits began triggering page faults.
• The format and API are not frozen, the software is changing quickly, and instabilities are possible.
• Vision is supported but expensive: a single 896×896 image consumes 256 prompt positions and takes about 2.8 seconds per image position after the initial vision-tower pass.

Verdict

WASTE is worth watching if you believe frontier models should run locally on consumer hardware, or if you want an embeddable, dependency-free C engine for MoE inference. If you need fast token generation or a stable API today, look elsewhere.