Baidu open-sources ERNIE 4.5, a 424B-parameter MoE model family

What it does

ERNIE 4.5 is a family of ten language and vision-language models built on a Mixture-of-Experts architecture. The largest variant packs 424 billion total parameters with 47 billion active, while the smallest is a 0.3 billion dense model. The repository also ships ERNIEKit, an industrial-grade toolkit for supervised fine-tuning, LoRA, and inference built atop the PaddlePaddle framework.

The interesting bit

The architecture uses a heterogeneous MoE structure that shares some parameters across text and vision while keeping others modality-specific, aiming to improve multimodal reasoning without degrading text performance. The team reports hitting 47% Model FLOPs Utilization during pre-training of the largest language model—a concrete throughput figure in an era of vague efficiency claims.

Key highlights

• The 300B-parameter base model is reported to beat DeepSeek-V3-671B on 22 of 28 benchmarks, while the 21B model claims wins over Qwen3-30B on math and reasoning tasks.
• Vision-language variants handle text, image, and video inputs with both thinking and non-thinking modes; the largest VL model is said to narrow or surpass the gap to OpenAI-o1 on visual reasoning benchmarks.
• ERNIEKit supports SFT, function-call training, padding-free batching, and 4-bit/2-bit quantization via convolutional code quantization.
• All weights and inference code are released on Hugging Face under Apache 2.0.
• The stack targets multi-hardware compatibility, including support for iluvatar GPUs.

Caveats

• The entire stack is tightly coupled to the PaddlePaddle ecosystem; if your infrastructure is PyTorch-native, the migration path is unclear.
• Benchmark comparisons in the README are self-reported by the team; independent verification is not discussed.

Verdict

Worth exploring if you need a fully open, large-scale MoE family with training infrastructure inside the PaddlePaddle orbit. Give it a pass if you require a framework-agnostic or PyTorch-native toolchain.