Meta's trick for running Llama on your phone without melting it

What it does ExecuTorch compiles PyTorch models ahead-of-time into .pte files, then runs them via a tiny C++ runtime that spans microcontrollers to smartphones. You export once with torch.export(), swap a partitioner line to target Qualcomm, Apple Neural Engine, or ARM Ethos-U, and ship. Meta uses this for Instagram, WhatsApp, Quest 3, and Ray-Ban Smart Glasses.

The interesting bit The runtime is small enough (50KB base) that “edge” isn’t a euphemism for “cloud fallback.” Partitioners automatically delegate subgraphs to NPUs or GPUs, falling back to CPU for whatever’s left. The same model graph runs across Android, iOS, embedded RTOS, and desktop—no ONNX or TFLite detours.

Key highlights

• Native PyTorch export; no format conversions or manual C++ rewrites
• 12+ hardware backends including CoreML, Qualcomm QNN, Vulkan, and ARM Ethos-U
• Built-in quantization via torchao (8-bit, 4-bit, dynamic)
• LLM-specific runners for Llama, Qwen, Phi-4-mini with streaming token generation
• Selective build strips unused operators to shrink binaries further

Caveats

• Several backends marked experimental or deprecated (CUDA, Metal, MPS)
• Desktop GPU support is explicitly not the focus; “see Desktop Guide” suggests it’s a secondary concern
• Embedded setup requires platform-specific build steps beyond pip install

Verdict Worth evaluating if you’re already in PyTorch and need on-device inference without maintaining separate mobile stacks. Skip if you’re heavily invested in TensorFlow Lite or need mature desktop GPU acceleration.