Xiaomi’s 7B model argues that reasoning is a data problem

What it does

MiMo-7B is a family of language models trained from scratch by Xiaomi for reasoning tasks. The release includes four full checkpoints—Base, SFT, RL-Zero, and RL—along with a technical report arguing that a model’s reasoning ceiling is set during pre-training, not just by later RL tuning.

The interesting bit

The team pre-trained on roughly 25 trillion tokens using a data pipeline specifically filtered to boost “reasoning pattern density,” then added Multiple-Token Prediction as an auxiliary objective. During post-training they use only rule-based accuracy rewards—no neural reward models—and counter sparse code rewards with a test-difficulty-driven scoring scheme.

Key highlights

• Four open checkpoints letting researchers inspect the full base→SFT→RL pipeline
• RL model scores 95.8 on MATH500 and 68.2 on AIME 2024 (Pass@1), with a newer 0530 revision reaching 80.1—surpassing DeepSeek R1’s 79.8 on that benchmark
• Custom inference support via vLLM and SGLang, including MTP layers for speculative decoding at ~90% acceptance rate
• “Seamless Rollout Engine” claims 2.29× training speedup via continuous rollouts and asynchronous reward computation

Caveats

• General benchmarks (MMLU-Pro, IF-Eval) lag significantly behind larger models and o1-mini, so MiMo is a narrow reasoning specialist
• The README describes training infrastructure and speedups, but the visible portion does not include code or architectural details for the rollout engine

Verdict

Worth exploring if you study small-model reasoning or need an open, end-to-end RL training trajectory to analyze. Skip it if you want a general-purpose chat model—MiMo’s strengths are strictly in math and code.