Async RL that scales from one GPU to a thousand

What it does

PRIME-RL is a reinforcement-learning training stack that handles the full post-training pipeline—SFT, RL, and evals—across anything from a single RTX 3090 to 1000+ GPU clusters. It uses FSDP2 for training and vLLM for inference, with explicit support for Slurm and Kubernetes deployments.

The interesting bit

The framework is built around asynchronous RL: the inference engine generates trajectories while the trainer updates weights, neither waiting for the other. For large MoE architectures—GLM-5, Qwen3 MoE, Nemotron H, GPT-OSS, and others—it ships custom kernels with expert parallelism and context parallelism, falling back to standard HuggingFace implementations only when no optimized path exists.

Key highlights

• Native integration with the verifiers environment hub, including SWE and agentic tasks
• FP8 inference, PD disaggregation, and multimodal support for Qwen3-VL
• LoRA training without SFT warmup, plus trajectory merging and pluggable advantage filters
• Examples range from “reverse text on one GPU” to reproducing the full INTELLECT-3.1 training run

Caveats

• Requires at least one NVIDIA GPU; no CPU-only path
• Flash Attention 3 builds from source on Hopper and is finicky with uv reinstalls
• GPT-OSS MoE models get context parallelism but not expert parallelism in the current release

Verdict Worth a look if you’re doing large-scale RL on modern MoE models or need the async throughput for agentic environments. Skip it if you’re on AMD, CPU-only, or just fine-tuning small dense models with standard tools.