Hand-rolled CUDA kernels for when FlashInfer isn't fast enough

What it does

HPC-Ops is a C++ and Python library of hand-optimized CUDA kernels for LLM inference, built by Tencent’s Hunyuan AI Infra team. It covers the usual suspects—attention, grouped GEMM, MoE fusion, sampling, and normalization—with a clean Python API meant to drop into existing frameworks like vLLM and SGLang. The kernels target NVIDIA Hopper-class GPUs and are benchmarked against FlashInfer, TensorRT-LLM, and others.

The interesting bit

The project reads like a cookbook of modern CUDA tricks rather than a black-box SDK. It decomposes FP32 weights into dual BF16 Tensor Core GEMMs for accuracy-sensitive router math, fuses AllReduce with RMSNorm over NVLink, and uses greedy bin-packing to schedule decode attention tiles across CTAs at runtime. It also doubles as a tutorial for CuTe, CUTLASS, TMA, and PDL in compact, production-grade form.

Key highlights

• Dynamic decode attention rebalances mixed-length request tiles across CTAs at runtime, claiming up to 2.88× over static split-k scheduling.
• FP8 block-sparse prefill attention skips masked KV tiles entirely with per-tile scaling, claiming up to 3.16× over dense baselines.
• A fused sampler collapses temperature scaling, top-k/top-p, softmax, and random sampling into two kernels, claiming up to 8.5× over fragmented PyTorch-style pipelines.
• Fused AllReduce + RMSNorm kernels use CUDA multicast or Lamport P2P depending on prefill vs. decode shape, claiming up to 1.76× over NCCL-based paths.
• BF16 × FP32 GEMM preserves FP32 accuracy via dual Tensor Core passes without falling back to CUDA cores, claiming up to 3.22× over cuBLAS FP32.

Caveats

• Hardware lock-in is explicit: the kernels require NVIDIA SM90 (Hopper/H20) and CUDA 12.8+, so older GPUs need not apply.
• The README notes performance “varies across cases,” and the benchmark table is dense with specific shapes and baselines that may not map cleanly to your stack.
• It is fundamentally a kernel library, not a full inference engine; you still need a framework to orchestrate the model.

Verdict

Worth a look if you operate LLM serving at scale on Hopper hardware and need to shave milliseconds off attention, MoE, or communication paths. Skip it if you are on Ampere or older, or if you are looking for a batteries-included inference server rather than drop-in operators.