JD.com’s inference engine for China’s homegrown AI chips

What it does xLLM is a C++ inference engine for LLMs, VLMs, and multimodal models that targets domestic AI accelerators—specifically NPU (A2, A3), MLU, ILU (BI150), and MUSA (S5000). It separates the serving layer from the execution engine to support elastic online/offline scheduling, dynamic PD disaggregation, and multimodal fault tolerance. JD.com deploys it in production for customer service, risk control, supply chain optimization, and ad recommendation.

The interesting bit xLLM is unusual in targeting domestic accelerators rather than mainstream NVIDIA hardware. It pragmatically borrows proven components—graph construction inspired by ScaleLLM and global KV-cache management built directly on Mooncake—rather than reinventing the wheel.

Key highlights

• Day-0 support for recent models including DeepSeek-V4, GLM-5, and Qwen3
• Hardware targets NPU (A2/A3), MLU, ILU (BI150), and MUSA (S5000)
• Service-engine decoupled architecture with speculative decoding and dynamic MoE expert load balancing
• Hierarchical KV cache offloading and prefetching via a Mooncake-based hybrid backend
• Battle-tested in JD.com’s core retail businesses

Caveats

• The hardware support table lists MLU with no example or driver details, suggesting maturity varies by platform
• No performance benchmarks or throughput numbers are published in the README
• Community channels and documentation lean China-centric (WeChat QR codes, Chinese-language docs)

Verdict Evaluate it if you are deploying on Chinese domestic accelerators and want a framework already hardened inside a major retail production environment. If your stack is standard NVIDIA-only infrastructure, this project is not aimed at you.