Learning CUDA by rewriting cuBLAS and FlashAttention

What it does

LeetCUDA is a curriculum of 200+ CUDA kernels and 100+ blog posts that starts at beginner level and quickly escalates to hand-rolled HGEMM and FlashAttention-2 implementations using raw MMA PTX. Every kernel ships with PyTorch bindings so you can benchmark your homework against actual library code. The premise is that peak GPU performance is the final exam, not just a chapter heading.

The interesting bit

The author treats cuBLAS not as a ceiling but as a classmate to beat: their toy HGEMM hits 98%–100% of its TFLOPS on several GPUs. The FlashAttention-2 variants are equally aggressive, using shared QKV memory, prefetching, and fine-grained tiling to outrun the reference in small-batch regimes—though the README candidly admits large-scale attention still trails.

Key highlights

• HGEMM kernels built with WMMA, MMA, and CuTe that reportedly reach 98%–100% of cuBLAS performance on hardware like the RTX 4090 and L20.
• A flash-attn suite written in pure MMA PTX, exploring six memory and tiling strategies from naive split-KV to fully shared QKV and fine-grained tilings that cut SRAM complexity.
• Small-batch attention benchmarks (D=64) showing up to ~1.5× speedup over FlashAttention-2 on an RTX 3080 Laptop, while the related ffpa-attn project hits ~1.8× over SDPA on L20 for larger head dimensions.
• Over 200 ranked kernels (Easy to Hard++) and 100+ LLM/CUDA write-ups, making it a structured reference rather than a scattered gist collection.
• Precision errors against standard implementations are kept tight, with a reported maximum below ~1e-3.

Caveats

• The performance wins are currently confined to small-scale attention (batch ≤ 4, heads ≤ 48, sequence length ≤ 8192, head dim ≤ 64); the README explicitly notes a remaining gap for large-scale workloads.
• Some variants use FP16 MMA accumulation while the reference uses FP32, so check the precision trade-offs before dropping these into a training pipeline.

Verdict

Bookmark this if you learn best by taking library code apart and rewriting it from scratch. Look elsewhere if you need battle-tested, drop-in replacements today; this is a teaching dojo, not a product.