Cross-platform LLM inference that landed inside Alibaba’s MNN

What it does mnn-llm is a C++ deployment layer for open large language models built on Alibaba’s MNN inference engine. It packages models such as Qwen, ChatGLM, Llama, Baichuan, and DeepSeek for offline inference across Linux, macOS, Windows, Android, and iOS. The maintainers note that the codebase has already been merged into the upstream MNN repository.

The interesting bit Rather than hand-writing per-model GPU kernels, the project leans on MNN’s existing backends — CUDA, OpenCL, Metal, or plain CPU — toggled at compile time. It also provides CLI, web, Android, and iOS chat demos, with the iOS example proudly labeled as entirely ChatGPT-generated.

Key highlights

• Broad model coverage, including vision-language variants (Qwen-VL, Qwen2-VL) and text-embedding models (bge, gte).
• Cross-platform builds with backend abstraction: the same source targets phones, desktops, and edge devices.
• Optional vision support is controlled by a compile flag (LLM_SUPPORT_VISION), keeping base builds lean.
• Companion export tool (llm-export) handles the ONNX-to-MNN conversion pipeline.

Verdict Worth a look if you need to ship quantized LLMs on mobile or embedded hardware inside the MNN ecosystem. Skip it if you are already committed to llama.cpp or MLX and have no interest in Alibaba’s toolchain.