Phone agents don't need a VLM to tap the same button 10,000 times
PhoneCLI compiles an app's navigation into deterministic commands so routine phone tasks replay in sub-second time with zero model calls.

What it does
PhoneCLI (repo name PhoneGo, from HKUDS) is a mobile agent harness that stops treating every task as novel GUI exploration. Offline, a BFS crawler walks an app’s screens and records them into a YAML “app map” — screens, tappable elements, and the edges between them. Online, a natural-language task is routed to a compiled macro with a single text-only call, then replayed deterministically via WebDriverAgent (iOS) or ADB (Android). Anything the map doesn’t cover falls back to a VLM loop, so compilation can only help, never hurt.
The interesting bit
The argument is that the bottleneck in phone agents was never model size — it’s that most of what a screenshot→VLM→tap loop does is navigation, which is static and repeated endlessly. The fix is the same reason CLIs beat GUIs: precompute the graph once, replay it forever. A routine task costs one text-only routing call plus at most one screenshot verification, instead of a VLM call per step.
Key highlights
- Three-tier serving: CLI → on-device 3B model → cloud; the CLI tier absorbs navigation at zero model cost.
- On AndroidLab (9 apps, 138 tasks): 63.0% vs 50.7% success — +12.3 points over the pure-VLM baseline, with 11% fewer steps and 14% fewer tokens.
- Compiled commands execute with 0 model calls, sub-second, fully on-device.
- Ships pre-built maps for 8 apps (~400 screens, 6,366 elements total); the crawler caps at 50 screens per app.
- The fallback path takes any model — the open 3B on-device model is an engine, not a requirement.
Caveats
- App maps are per-app and one-time work (~10 minutes each per the README); apps outside the shipped 8 need mapping before they benefit.
- The headline ~10% cloud-call reduction is modest next to the benchmark numbers; the big wins come from the AndroidLab evaluation setup.
- iOS support leans on WebDriverAgent, which carries its own setup weight (documented separately).
Verdict
Worth a look if you build mobile/GUI agents and are tired of paying a VLM to find the Settings page. If your tasks are genuinely novel every time, the compiled tier won’t earn its keep — but the fallback design means you lose nothing by trying.
Frequently asked
- What is HKUDS/PhoneGo?
- PhoneCLI compiles an app's navigation into deterministic commands so routine phone tasks replay in sub-second time with zero model calls.
- Is PhoneGo open source?
- Yes — HKUDS/PhoneGo is open source, released under the MIT license.
- What language is PhoneGo written in?
- HKUDS/PhoneGo is primarily written in Python.
- How popular is PhoneGo?
- HKUDS/PhoneGo has 1.4k stars on GitHub and is currently accelerating.
- Where can I find PhoneGo?
- HKUDS/PhoneGo is on GitHub at https://github.com/HKUDS/PhoneGo.