Alibaba's pocket-sized engine that runs LLMs on your phone

What it does MNN is a C++ inference and training engine built for resource-starved devices. It converts models from TensorFlow, ONNX, PyTorch and others into a compact runtime that executes on ARM, x86, Metal, OpenCL, Vulkan and CUDA. The core Android library weighs roughly 800 KB; iOS adds about 2 MB to your binary. Inside Alibaba it powers Taobao, Tmall, Youku and roughly 70 other scenarios ranging from live-stream filters to fraud detection.

The interesting bit The project has quietly pivoted from classic CNN acceleration to becoming a full-stack LLM runtime. MNN-LLM now runs Qwen, DeepSeek-R1, LLaMA and others locally on Android, iOS and embedded Linux, while MNN-Diffusion does the same for Stable Diffusion. It is the compute kernel inside Walle, Alibaba’s OSDI'22-published device-cloud collaborative ML system, which means the optimizations are production-grade rather than benchmark-ware.

Key highlights

• Supports 178 TensorFlow ops, 163 TorchScript ops, 158 ONNX ops with graph optimization passes
• Winograd convolution for symmetric kernels (3×3 through 7×7), ARMv8.2 FP16 and VNNI/SDOT acceleration
• FP16 and Int8 quantization can shrink models 50–70%
• Ships with reference apps: multimodal chat (text/image/audio), 3D avatar with on-device ASR/TTS, and Sana image editing
• Python API and a NumPy-like tensor compute module for engineers who want to avoid C++

Caveats

• Community support is predominantly Chinese; English speakers may need patience
• BF16 support is patchy across backends, and “native” CPU precision is marked “B” (bugs or unoptimized) in the project’s own matrix
• NNAPI and QNN backends are listed as incomplete or unsupported

Verdict Worth a look if you are shipping on-device AI to Android, iOS or IoT and need a single runtime that spans LLMs, diffusion and classic vision models. Probably overkill if you are already happy with CoreML or only targeting NVIDIA servers.