Unbundling RL training and inference for omni-modal post-training

What it does

Relax is a reinforcement learning post-training framework built by the Xiaohongshu AI Infra Team for multimodal large language models. It uses Ray Serve to turn every stage—rollout, actor training, reference modeling, and advantage estimation—into independent services that run on separate GPU clusters. Data streams between them via TransferQueue, so inference and training never block each other, and the system supports text, vision, and audio in one pipeline.

The interesting bit

The architecture is aggressively service-oriented: every role is a Ray Serve deployment with its own placement group, elastic scaling, and fault recovery. This means you can grow or shrink inference capacity mid-training via a REST API, or let reference and advantage calculations run in-process on the actor’s own GPUs in Hybrid mode to cut idle time. It treats a GPU cluster less like a monolithic supercomputer and more like a load-balanced microservice mesh.

Key highlights

• End-to-end omni-modal RL for text, vision, and audio—including Qwen3-Omni post-training
• Three execution modes: synchronous colocation, fully async multi-cluster, and a Hybrid mode that mixes streaming data with in-process forward passes
• Built-in algorithms: GRPO, GSPO, SAPO, CISPO, and on-policy distillation, plus pluggable rewards and GenRM (LLM-as-judge)
• Elastic rollout scaling via HTTP REST API across same-cluster or federated external clusters
• Production ops: health-managed auto-recovery, metrics export, and real-time notifications

Verdict

Teams already running Megatron-LM and Ray clusters for multimodal model post-training should evaluate this immediately; solo researchers or anyone looking for a lightweight, single-GPU RLHF script will find it overwhelming.