Read the source of ten modern LLMs, one PyTorch repo at a time

What it does This repo reimplements a decade of LLM architecture evolution in plain PyTorch: GPT-2 XL, Llama 3 and 3.2, OLMo 2, Phi-4, DeepSeek v3, Gemma 3, xLSTM, Mistral 3.1 Small, and Llama 4 Maverick. Each model lives in its own directory, so you can diff the designs side by side — where LayerNorm became RMSNorm, where absolute positional embeddings gave way to RoPE, where attention got grouped and windows started sliding. Qwen3 (32B and the 235B MoE variant) is listed as in progress.

The interesting bit The comparison table is the real product: it turns ten repos into a mini-history of the field. You can see, in one glance, that nearly everything since Llama 3 converged on RMSNorm + RoPE + GQA, and that the interesting divergence is elsewhere — sliding-window attention in Gemma 3 and Mistral, mLSTM in xLSTM, and mixture-of-experts in DeepSeek v3 and Llama 4.

Key highlights

• Ten architectures implemented from scratch, spanning 1.5B (GPT-2 XL) to 671B (DeepSeek v3) parameter scales
• Covers multimodal models too: Gemma 3 (27B) and Mistral 3.1 Small (24B) handle text plus image
• Includes xLSTM, the non-transformer outlier, using mLSTM cells
• MoE implementations for DeepSeek v3 and Llama 4 Maverick (400B)
• Qwen3 variants (32B, 235B-A22B) on the roadmap

Caveats

• The README doesn’t say whether these implementations are trained, load real weights, or are forward-pass reference code only — that’s left unclear.
• No tests, docs, or benchmarks are mentioned, so expect to read the code itself as the documentation.

Verdict Great for anyone who wants to understand modern LLM internals by reading working code rather than papers. If you’re looking for something to run in production, this isn’t that — it’s a learning artifact, and a well-organized one.