An FSDP Trainer That Outruns 3D Parallelism on 200B MoEs

What it does

XTuner V1 is a training engine for ultra-large Mixture-of-Experts models, scaling up to 1T parameters. It covers pre-training, fine-tuning, and reinforcement learning—GRPO is already implemented—for dense and MoE architectures like Qwen3 MoE, DeepSeek V3, and Intern S1, with specific optimizations for Ascend NPUs alongside NVIDIA GPUs. Multimodal pre-training and supervised fine-tuning for vision-language models are also bundled in.

The interesting bit

Instead of defaulting to traditional 3D parallelism, XTuner V1 bets on FSDP and “dropless” strategies: it claims you can train a 200B MoE without any expert parallelism, and a 600B model with only intra-node expert parallelism. It also says 200B models can run 64k sequence lengths without sequence parallelism, which is either a serious memory-engineering feat or a very specific benchmark configuration.

Key highlights

• Claims first FSDP throughput to surpass traditional 3D parallel schemes on MoE models above 200B scale.
• “Dropless” training: 200B MoE without expert parallelism, 600B with intra-node only.
• 200B MoE models on 64k sequences sans sequence parallelism, plus optional DeepSpeed Ulysses scaling.
• Dual hardware target: NVIDIA GPUs (FP8/BF16) and Ascend NPUs (BF16), with a roadmap favoring NPU optimization.
• Inference through LMDeploy; vLLM and SGLang integrations are still on the todo list.

Caveats

• NPU support is missing for GPT OSS, DeepSeek V3, and KIMI K2 (marked 🚧).
• Algorithmic additions like MPO, DAPO, and multi-turn agentic RL are listed as coming soon, not shipped.
• Inference integrations beyond LMDeploy remain unfinished.

Verdict

Worth evaluating if you’re training large MoEs and want to sidestep the full complexity of 3D parallelism—particularly on Ascend hardware. If you need mature inference across vLLM or SGLang today, it is not yet a turnkey stack.