A GPT small enough to fit in a single C file

What it does

MicroGPT-C is a minimal, self-contained GPT implementation in a single C file. It trains and runs inference without touching a single external library—no BLAS, no OpenMP, not even a build system beyond gcc. The README shows it learning to generate names from a small dataset, dropping loss from 3.3 to ~2.2 over 20,000 steps, then spitting out plausible-sounding outputs like “kayley” and “maria.”

The interesting bit

The project treats “dependency-free” as a feature, not a constraint. It hand-rolls matrix multiplications with AVX2 intrinsics and links only against libm for sqrt and exp. The README claims 2.6 million tokens per second in FP32 on CPU, which—if accurate—suggests the SIMD kernels are tight enough to compete with naive PyTorch CPU inference. Whether that number holds on non-AVX2 hardware is unclear from the sources.

Key highlights

• Single-file C implementation: no headers to chase, no CMake to fight
• Explicit compiler flags with explanations (-O3, -march=native, -ffast-math)
• Hand-vectorized FP32 kernels using AVX2
• Trains from scratch on a toy name-generation task
• Claims ~2.6M tok/sec throughput on CPU inference

Caveats

• README is extremely sparse: no architecture details, no model size, no dataset description beyond the sample output
• The 2.6M tok/sec figure lacks hardware context or reproducibility notes
• -ffast-math trades IEEE compliance for speed; training stability implications unmentioned

Verdict

Worth a look if you want to dissect a transformer without navigating 10,000 lines of Python indirection, or if you’re teaching someone how attention actually maps to memory layouts. Skip it if you need pretrained weights, multi-GPU scaling, or any of the comforts of a modern ML framework.