Google's LLM runtime that actually ships in Chrome and watches

What it does

LiteRT-LM is Google’s inference framework for squeezing large language models onto edge hardware. It handles model execution across CPU, GPU, and NPU backends, with bindings for Python, Kotlin, C++, Swift (early preview), and even browser JavaScript. The project also includes a CLI tool for running models directly without writing code.

The interesting bit

This isn’t experimental—it’s already running inside Chrome, Chromebook Plus, and Pixel Watch. The framework supports multi-token prediction (MTP) for speculative decoding, which Google claims makes Gemma 4 inference up to 3× faster. It also handles vision and audio inputs, plus function calling for agentic workflows, all without leaving the device.

Key highlights

• Ships in actual Google products, not just demo apps
• Cross-platform: Android, iOS, web, desktop, and IoT including Raspberry Pi
• Hardware acceleration via GPU and NPU across Linux, macOS, and Windows
• Multi-modality support (vision, audio) and tool use / function calling
• Broad model compatibility: Gemma, Llama, Phi-4, Qwen, and others
• Community Flutter package available via flutter_gemma

Caveats

• Swift and JavaScript APIs are marked “early preview,” not stable
• Flutter support is community-maintained, not first-party
• The README is heavy on feature lists and light on technical architecture details

Verdict

Mobile and embedded developers who need on-device LLM inference without reinventing the wheel should look here. If you’re running everything in the cloud anyway, or need training capabilities, this is the wrong layer of the stack.