AI agents shouldn’t need computer vision to click a button

What it does

agent-desktop is a native Rust CLI that exposes the macOS accessibility tree as a structured JSON API for AI agents. It can list apps, snapshot UI elements, click buttons, type text, press keys, manage windows, and read the clipboard—all by referencing deterministic element IDs like @e3 rather than screen coordinates. The output is machine-readable JSON with error codes and recovery hints, designed to be consumed by an agent loop of observe-decide-act.

The interesting bit

Instead of scraping screenshots or simulating mouse coordinates, it traverses the OS accessibility tree directly, which yields a 78–96% token reduction on dense apps like Slack or VS Code when using progressive skeleton traversal. It also ships as a C-ABI dynamic library so you can load it in-process from Python, Swift, or Node without forking a process per command.

Key highlights

• 54 commands covering observation, interaction, keyboard, mouse, clipboard, window management, and notifications.
• Progressive skeleton traversal: shallow overview plus targeted drill-down to cut token usage on complex UIs.
• Headless-by-default interactions that block unwanted focus, cursor, or pasteboard side effects.
• Deterministic element references (@e1, @e2) and snapshot IDs for stable, repeatable agent workflows.
• Prebuilt FFI library (libagent_desktop_ffi) for in-process integration with most major languages.

Caveats

• macOS 13.0+ is required, and the tool needs Accessibility (and Screen Recording for screenshots) permissions granted manually.
• Notifications support is macOS-only; while FFI artifacts ship for Linux and Windows, the documentation and CLI examples are overwhelmingly macOS-centric.

Verdict

Worth a look if you are building an AI agent that needs to interact with native desktop applications reliably without computer vision. Skip it if your automation targets are web-only or you are not prepared to wrangle macOS accessibility permissions.