Your RPA Scripts Break Because Buttons Move

What it does

AskUI Python SDK hands the keyboard and mouse to an AI agent that navigates software by looking at the screen. You describe the goal in plain English—open a browser, find a login form, check an account balance—and the SDK translates that into clicks, scrolls, and keystrokes on Windows, macOS, Linux, or Android. It handles both granular single-step commands and open-ended agentic tasks, and it can return structured data extracted from the UI.

The interesting bit

Because it locates elements by vision and language rather than by DOM or accessibility trees, the automation survives reskins, repositioned buttons, and cross-platform UI differences. A built-in caching layer keeps repeated identical screens from triggering another round of expensive vision-model API calls.

Key highlights

• Finds controls by text, image, or natural-language description instead of brittle selectors.
• Swappable model backends: Anthropic Claude, Google Gemini, Hugging Face, OpenRouter, or AskUI’s own hosted option.
• Extensible tool ecosystem via Model Context Protocol for custom actions like screenshot archiving or console logging.
• Optional install extras for AWS Bedrock, Google Vertex, OpenTelemetry, Playwright web flows, and Office document parsing.
• Built-in reporting, telemetry tracing, and structured data extraction for test-automation pipelines.

Caveats

• The Python package alone cannot drive the cursor; you still need to install the separate AskUI Agent OS runtime.
• Using the hosted AskUI models requires signing up for a workspace and exporting an access token.
• iOS is mentioned in the repository description but is not listed among the explicitly supported platforms in the README.

Verdict

This is aimed at QA engineers and RPA builders who are tired of repairing broken selectors after every frontend update. Skip it if your workflow is already fully API-driven or confined to headless shells.