Android in a browser tab for agents who shouldn’t touch your wallet

What it does

MobileGym runs a simulated Android environment inside a browser, complete with 28 apps and 416 parameterized tasks. The entire UI state is exposed as a structured JSON snapshot, so judges can verify agent actions programmatically in sub-millisecond time instead of asking a VLM to squint at screenshots. It is designed to spin up hundreds of lightweight, isolated instances on a single server for parallel rollouts and online RL.

The interesting bit

The project tackles the three hard walls of real-device agent research: unreadable app state, unwritable databases, and irreversible side effects like money transfers. Because the environment is fully programmable, you can snapshot, clone, and reset state at will—then validate that simulation gains actually transfer to physical hardware (the authors report 95.1% retention on a Redmi Note 12 Turbo).

Key highlights

• Deterministic, code-level judges eliminate VLM misjudgment; no string-similarity guesswork needed.
• Runs 256 parallel instances on one server at roughly 400 MB RAM and about 3 seconds cold-start per instance.
• Ships with an AnswerSheet protocol that forces agents to fill structured forms, preventing chain-of-thought leakage from gaming free-text metrics.
• Apps are built as finite-state machine specs, so screens and transitions can be traversed by static analysis or trajectory search.
• Sim-to-real validated: a GRPO run on Qwen3-VL-4B gained +42.8 points in simulation and +40.7 points on a real device.

Caveats

• The companion dataset carries a CC BY-NC 4.0 license, so commercial use of the bundled synthetic data is off-limits.
• It simulates Android via declarative manifests rather than running a full OS stack, which means behavioral fidelity is the goal, not pixel-perfect emulation.

Verdict

MobileGym is a strong fit for RL and agent researchers who need reproducible, large-scale evaluation on stateful mobile tasks without risking real accounts or money. If you are looking for a generic Android emulator to run arbitrary APKs, this is not it.