Teaching LLMs to tap, swipe, and scrape on real devices

What it does

mobile-use is an open-source Python agent that drives Android devices and iOS simulators via natural language. It peers into the screen through the OS accessibility tree, delegates planning to an LLM, and executes taps and swipes to complete tasks ranging from sending messages to scraping app data into structured JSON. The whole orchestration runs on a LangGraph multi-agent architecture instead of a single brittle prompt chain.

The interesting bit

The README highlights a 100% score on the AndroidWorld benchmark and an arXiv paper to match, suggesting the task-decomposition approach is more than vaporware. A nice touch: the system architecture diagram is auto-generated from the codebase itself, so the docs stay honest as the graph evolves.

Key highlights

• Drives physical Android phones over USB, Android emulators, and iOS simulators on macOS.
• Extracts structured data from apps using plain-English descriptions of the desired output shape.
• Swappable LLM backends via config files, including local models and OpenAI-compatible endpoints.
• Uses multiple coordinated agents rather than one monolithic model call.
• Apache 2.0, with a polite request to cite Minitap if you reuse the architecture.

Caveats

• Games and graphics-heavy apps are largely off-limits because the agent depends on accessibility tree data those surfaces rarely expose.
• Physical iOS devices remain unsupported; only simulators on macOS work today.
• The turnkey Docker quickstart is Android-only, so iOS setups require more manual lifting.

Verdict

Worth a look if you are building QA automation over apps that lack APIs, or researching screen-reading agents that need to prove their benchmark scores. Skip it if your target is mobile games or you need a zero-friction solution for physical iPhones.