Llama 2 inference distilled into one 700-line C file

What it does

Train a Llama 2 architecture model in PyTorch, then run inference through a single ~700-line C file called run.c. The author provides pre-trained “baby” models—15M to 110M parameters trained on TinyStories—that generate coherent short stories, and the code can ingest Meta’s official Llama 2 checkpoints up to 7B after export. An optional quantized path in runq.c adds int8 weight and activation quantization to shrink checkpoints and speed up integer matmuls.

The interesting bit

This is a deliberate exercise in radical minimalism: the architecture is hard-coded, there are zero dependencies, and the code is meant to be educational rather than competitive with production engines like llama.cpp. It began as a weekend offshoot of nanoGPT and asks “how simple can this be?” instead of “how fast can this go?”

Key highlights

• Single-file inference in pure C (run.c) with no external libraries.
• The 15M-parameter baby model runs at ~110 tokens/s on an M1 MacBook Air.
• Can export and run Meta’s Llama 2 7B base and chat models; partial, buggy support for Code Llama exists.
• Optional int8 quantization (runq.c) using Q8_0 cuts a 7B checkpoint from 26GB down to 6.7GB.
• The project is young, moves quickly, and wears its “fun weekend project” origins openly.

Caveats

• Anything above 7B parameters is effectively off-limits: 13B+ currently fails due to “integer flow in pointer arithmetic,” and even if fixed, the author notes that fp32 inference would be “fairly unusably slow.”
• Code Llama support is incomplete because some hyperparameters differ from Llama 2, so inference is “not exactly correct and a bit buggy.”

Verdict

Reach for this if you want to understand transformer inference by reading a single C file, or if you enjoy watching tiny models tell stories about sharing teapots. Skip it if you need production-grade throughput or reliable support for models above 7B parameters.