One agent, every screen you own — including the phone in your pocket

What it does AutoCua is a Python harness for computer-use agents. One mode string picks the surface — desktop, terminal, your own Chrome, an iOS Simulator or a real iPhone over USB, or an Android emulator or phone — and the same agent clicks, types, browses, and writes code. It supports 48 models across 9 providers, so you’re not locked to one vendor’s CUA offering.

The interesting bit The model never sees coordinates. It picks numbered elements from a view fusing the OS accessibility tree with OCR, and a number it wasn’t shown is rejected before anything moves. Input is injected at the OS’s own input layer (kCGHIDEventTap on macOS, pywinauto on Windows, a kernel driver when UAC blocks everything else), so apps see events indistinguishable from your hand. A separate compression agent splices handoff summaries into the transcript mid-run, so long tasks don’t die at the context limit.

Key highlights

• Real hardware input events on macOS, Windows, and Linux — no synthetic-coordinate clicking
• Drives your own Chrome via an extension, with Chrome’s visible “AutoCua started debugging this browser” bar and a Cancel button; a debugging port opens only for headless runs
• Real phones over USB (or Wi-Fi via the desktop app), plus simulators and emulators
• Rust browser and Android agents ship prebuilt in the main wheels
• No telemetry — nothing leaves except to your chosen model provider (and Telegram, if you wire it up yourself)

Caveats

• Linux needs GNOME (Wayland tested, X11 not) and a stack of distro packages pip can’t install; the Windows UAC kernel driver is a manual, reboot-requiring step — without it, tasks stop at elevation prompts
• The Interception driver is dual-licensed LGPL-3.0 non-commercial / paid commercial, so it’s deliberately not bundled
• iOS mobile use needs a macOS host with full Xcode

Verdict Worth a look if you want a model-agnostic CUA that treats real input and accessibility trees as first-class, and covers mobile as well as desktop. Skip it if you want a zero-setup, single-command experience — the plumbing is honest about what pip can’t do.