A field manual for programming Blackwell GPUs at the IR level

What it does

It is an open online book that teaches GPU kernel programming by treating the Blackwell-class GPU as the central subject. The material progresses from hardware fundamentals—execution models, memory hierarchy, and asynchronous engines—through a Python DSL called TIRx, and culminates in production-grade kernels for GEMM and Flash Attention 4.

The interesting bit

Instead of stopping at abstract API calls, the text builds a tiled GEMM step-by-step into a state-of-the-art implementation using TMA pipelining, persistent scheduling, and warp specialization, then reuses those primitives to assemble a complete Flash Attention 4 kernel. TIRx itself is an unusual vehicle: it parses Python source via inspection to generate GPU IR through Apache TVM, letting you write low-level kernels in a high-level syntax.

Key highlights

• Deep dive into Blackwell specifics: Tensor Memory, TMA, Tensor Cores, warpgroups, clusters, and Cluster Launch Control scheduling.
• Complete walkthrough from roofline analysis and data layout to a multi-CTA clustered GEMM.
• Full Flash Attention 4 coverage including two MMAs with softmax, online-softmax rescaling, causal masking, and GQA.
• TIRx language reference and compiler internals included for readers who want to extend the toolchain.
• Published in English and Chinese as a Sphinx site with auto-deployment.

Caveats

• Every runnable kernel targets sm_100a; you need a Blackwell GPU such as a B200 to execute the examples, though reading costs nothing.
• TIRx parses kernel source via Python source inspection, so kernels must live in files or notebook cells and cannot be passed as inline strings.
• The full reference GEMM and Flash Attention 4 kernels live in a separate companion package not included in the main book.

Verdict

Read this if you are a systems or compiler engineer who wants to understand Blackwell silicon deeply enough to write kernels that match or beat vendor libraries. Look elsewhere if you need a generic GPU introduction or lack access to the specific hardware required to test your code.