Your Mac's GUI, now with an API for robots

What it does

This Swift-native macOS automation layer lets scripts and AI agents capture the screen, inspect UI elements, and drive applications through clicks, typing, scrolling, and hotkeys. It exposes identical capabilities through both a native CLI and an MCP server, so tools like Codex, Claude Code, and Cursor can orchestrate multi-step workflows by seeing and interacting with the desktop. A natural-language agent command chains these primitives into higher-level tasks such as opening an app and filling out forms.

The interesting bit

The tool defaults to background input delivery, targeting specific processes without forcing them to the foreground or stealing focus—handy for automating Safari while you keep working elsewhere. It also surfaces macOS accessibility APIs directly, letting you set text field values or invoke named actions like AXPress without synthetic mouse events. Shell completions are generated from the same Commander metadata that powers the help system, which is a small but telling sign the author actually uses the CLI.

Key highlights

• Pixel-accurate screenshots of windows, screens, and the menu bar with optional Retina 2x scaling
• Background-targeted input (click, type, hotkey) that avoids focus-stealing when the target process is known
• Direct accessibility primitives: set-value and perform-action for reliable automation without coordinate guessing
• MCP server mode exposes the same toolset to remote AI providers (OpenAI, Anthropic, Ollama, LM Studio, etc.) via the Tachikoma abstraction
• Structured JSON discovery of menus, menubar items, windows, and apps before any interaction happens

Caveats

• Requires macOS 15 Sequoia or later, plus explicit Screen Recording, Accessibility, and Event Synthesizing permissions
• The Node.js dependency is only for the MCP wrapper and build scripts; the core runtime is Swift, so Linux and Windows users need community rewrites (PeekabooX, PeekabooWin)

Verdict Worth a look if you are building agentic workflows on macOS and need more than headless browser automation. Skip it if you are not on macOS 15+ or if your automation targets are already covered by API-first tools.