Nine thousand lines to train trillion-parameter agents

What it does

Molt is a reinforcement-learning training framework built for researchers who want to iterate on agents rather than infrastructure. It uses Ray for async orchestration, vLLM for rollout, and NVIDIA AutoModel with FSDP2 for training, all in pure PyTorch. The entire RL codebase is roughly 9,200 lines, yet it handles fully-async, multimodal, multi-turn agentic RL at scales up to trillion-parameter MoE models. Rewards are just Python: write an Env or ChatAgent subclass, return a Result(reward=...), and the single-actor trainer consumes it without backend ceremony.

The interesting bit

The framework is “token-first”: token IDs, logprobs, action ranges, rewards, and multimodal tensors stay aligned from rollout through training in one format. That alignment, plus built-in importance-sampling correction for the inevitable train/rollout logprob drift during async weight swaps, means you can hack the model in plain PyTorch without worrying that vLLM’s generation kernels and FSDP2’s training kernels are silently diverging.

Key highlights

• ~9.2K LOC of RL code across three thin layers; the README claims you can read it in an afternoon.
• One Gymnasium-aligned agent API (Env.step() or ChatAgent.run()) covers graders, tools, VLM environments, and LLM-as-judge calls.
• Scales from 8B to 1T-class MoE on the same script using AutoModel + FSDP2 with TP, EP, CP, and Adam CPU offload.
• Supports REINFORCE, GRPO, PPO/GAE, on-policy distillation, and per-token or sequence-level IS correction for off-policy async rollout.
• Chat templates and assistant spans are derived from the model’s own template rather than hard-coded markers.

Caveats

• The HF Transformers path is explicitly a “non-preferred fallback” that only supports text + flash_attention_2 + packing, with no CP, EP, or TP; if your model lacks a native AutoModel class, you lose the advanced parallelism features.
• The muon optimizer is experimental and has “shown no consistent win over adam,” which remains the recommended default.
• The comparison table and line counts were measured by the authors on specific dates (2026-07-20 for Molt, 2026-06-16 for competitors), so treat the ~9.2K claim as self-reported.

Verdict

Grab this if you are an RL researcher who wants to modify training logic in PyTorch without navigating 60K+ lines of framework indirection. Skip it if you need a production-grade, broad-coverage RLHF platform with extensive YAML configuration and stable third-party backend support.