Training GPT-2 without the 245MB PyTorch tax

What it does

llm.c trains transformer models—specifically the GPT-2 and GPT-3 families—from scratch using only C and CUDA. The repository keeps a parallel PyTorch reference (train_gpt2.py) so every forward pass, gradient, and loss value can be cross-checked for correctness. There is a bleeding-edge GPU path in train_gpt2.cu, a simpler single-GPU fp32 variant, and even a ~1,000-line CPU reference in train_gpt2.c for those who want to read the algorithm without the hardware.

The interesting bit

The project treats education and speed as non-conflicting goals. The dev/cuda directory collects hand-written, documented kernels ranging from naive to optimized, while the mainline freely adopts vendor libraries like cuBLAS and cuDNN when raw performance matters. It is currently about 7% faster than PyTorch Nightly, which is either a testament to the code or an indictment of how much overhead a framework can hide.

Key highlights

• Ships with a unit-test harness that verifies C/CUDA outputs match the PyTorch reference loss-for-loss
• Supports multi-GPU and multi-node training via MPI and NCCL, with three different initialization strategies depending on your cluster’s Slurm and PMIx support
• Includes a tutorial breaking down a single LayerNorm implementation in C as a gateway to the rest of the stack
• Flash Attention via cuDNN is available but disabled by default because it inflates compile times from seconds to roughly a minute
• The CPU version is explicitly framed as a “GPU poor” demo; it will run on a MacBook but it will not go far

Caveats

• The simplified fp32 and CPU implementations are legacy checkpoints frozen early in the project; they are easier to read but lack the mixed-precision and multi-node capabilities of the mainline
• cuDNN-backed Flash Attention is described as “very new” and requires manual header wrangling to enable
• Multi-node training demands that you navigate Slurm and PMIx compatibility yourself, picking among three NCCL initialization methods

Verdict

This is for developers who want to see LLM training without framework magic, or who need a fast, hackable training substrate that still validates itself against PyTorch. If you are looking for a batteries-included, high-level API, this is deliberately not it.