← all repositories
HKUDS/PhoneGo

Phone agents don't need a VLM to tap the same button 10,000 times

PhoneCLI compiles an app's navigation into deterministic commands so routine phone tasks replay in sub-second time with zero model calls.

★1.4k stars Python Agents
PhoneGo
Velocity · 7d
+59
★ / day
Trend
↗accelerating
star history

What it does

PhoneCLI (repo name PhoneGo, from HKUDS) is a mobile agent harness that stops treating every task as novel GUI exploration. Offline, a BFS crawler walks an app’s screens and records them into a YAML “app map” — screens, tappable elements, and the edges between them. Online, a natural-language task is routed to a compiled macro with a single text-only call, then replayed deterministically via WebDriverAgent (iOS) or ADB (Android). Anything the map doesn’t cover falls back to a VLM loop, so compilation can only help, never hurt.

The interesting bit

The argument is that the bottleneck in phone agents was never model size — it’s that most of what a screenshot→VLM→tap loop does is navigation, which is static and repeated endlessly. The fix is the same reason CLIs beat GUIs: precompute the graph once, replay it forever. A routine task costs one text-only routing call plus at most one screenshot verification, instead of a VLM call per step.

Key highlights

  • Three-tier serving: CLI → on-device 3B model → cloud; the CLI tier absorbs navigation at zero model cost.
  • On AndroidLab (9 apps, 138 tasks): 63.0% vs 50.7% success — +12.3 points over the pure-VLM baseline, with 11% fewer steps and 14% fewer tokens.
  • Compiled commands execute with 0 model calls, sub-second, fully on-device.
  • Ships pre-built maps for 8 apps (~400 screens, 6,366 elements total); the crawler caps at 50 screens per app.
  • The fallback path takes any model — the open 3B on-device model is an engine, not a requirement.

Caveats

  • App maps are per-app and one-time work (~10 minutes each per the README); apps outside the shipped 8 need mapping before they benefit.
  • The headline ~10% cloud-call reduction is modest next to the benchmark numbers; the big wins come from the AndroidLab evaluation setup.
  • iOS support leans on WebDriverAgent, which carries its own setup weight (documented separately).

Verdict

Worth a look if you build mobile/GUI agents and are tired of paying a VLM to find the Settings page. If your tasks are genuinely novel every time, the compiled tier won’t earn its keep — but the fallback design means you lose nothing by trying.

Frequently asked

What is HKUDS/PhoneGo?
PhoneCLI compiles an app's navigation into deterministic commands so routine phone tasks replay in sub-second time with zero model calls.
Is PhoneGo open source?
Yes — HKUDS/PhoneGo is open source, released under the MIT license.
What language is PhoneGo written in?
HKUDS/PhoneGo is primarily written in Python.
How popular is PhoneGo?
HKUDS/PhoneGo has 1.4k stars on GitHub and is currently accelerating.
Where can I find PhoneGo?
HKUDS/PhoneGo is on GitHub at https://github.com/HKUDS/PhoneGo.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.