176 KB of C that runs a 2.78T-parameter model in 8 GB

What it does This is a portable C99 inference engine for Kimi K3, a 2.78-trillion-parameter mixture-of-experts transformer. It runs the full model on a single Linux x86-64 CPU with no BLAS, no GPU, and no framework dependencies. Depending on the preset, peak RSS sits at 8.24 GB or scales up to 224 GB, yet the generated output is byte-identical across every memory budget.

The interesting bit The engine treats the 1.56 TB checkpoint as a cold stream rather than a resident database. Only a configurable “trunk” of dense layers stays pinned in RAM; the 1.45 TB of routed experts live on disk in packed MXFP4 and are multiplied straight from their half-byte form without ever being fully dequantized. Four separate memory reductions—4-bit expert packing, KDA attention with a fixed recurrent state, MLA latent caching, and trunk streaming—turn a server-cluster floor into a laptop dial.

Key highlights

• 176 KB static binary built from seven C files, with no dependencies beyond a C99 compiler, libm, and OpenMP.
• 16 of 896 experts fire per token; the rest remain on disk.
• Attention memory never grows with context length thanks to a recurrent KDA state.
• Byte-exact parity against a PyTorch reference verified across all 93 layers.
• First-token latency is brutal because it may pull ~108 GB from disk; steady-state is merely slow.

Caveats

• Inference is glacial: 32.69 seconds per token at the 8.24 GB “laptop” preset, and 10.69 seconds per token even when peak RSS reaches 127.92 GB.
• This is a base model with no chat template, so it continues text rather than holding a conversation.
• You still need ~1.7 TB of local disk to host the checkpoint and packed trunk.

Verdict Systems programmers and quantization researchers should study this as a masterclass in memory accounting and deterministic inference. If you need an interactive chatbot or real-time throughput, look elsewhere—this is an engineering demonstration, not a product.