One kernel launch for your entire LLM inference pipeline

What it does

Mirage Persistent Kernel (MPK) is a compiler and runtime that takes standard LLM inference and fuses it into a single GPU megakernel. That one kernel handles all computation and inter-GPU communication—such as all-reduce—inside a single launch, which the project claims cuts latency by 1.2× to 6.7×. You define the model graph in Python, chain fused operations like rmsnorm_linear_layer, and MPK generates the CUDA.

The interesting bit

The unusual part is collapsing an entire multi-GPU inference pipeline into one persistent kernel that manages its own workers and schedulers across streaming multiprocessors. Instead of the usual parade of separate kernel launches for each layer and communication step, everything runs inside a single grid where tasks are scheduled on the fly.

Key highlights

• Fuses computation and communication into one kernel launch, using nvshmem_tensor for remote GPU access during operations like all-reduce
• Targets multi-GPU setups with MPI-aware ranks and explicit worker/scheduler counts tied to physical SMs
• Exposes a Python API to define computation graphs and attach existing PyTorch tensors by name
• Ships with a demo compiling Qwen3-8B from Hugging Face into a megakernel
• Backed by an OSDI 2025 paper and a fresh arXiv preprint on megakernelization

Caveats

• Pre-built binary wheels are not yet available as of mid-June 2025; you currently build from source
• The number of workers and schedulers must exactly match the physical SM count, so configuration is hardware-specific
• The megakernel currently requires two specific meta tensors (step and tokens) to track decoding state

Verdict

Worth a look if you are chasing inference latency on multi-GPU nodes and don’t mind compiling from source while the project matures. Skip it if you need a polished, pip-installable toolchain today.