Stop writing selectors. Start describing goals.

What it does

Midscene.js turns natural-language goals into UI interactions on web pages, Android devices, iOS simulators, and even custom <canvas> surfaces. You write scripts in JavaScript or YAML, and the library delegates clicking, typing, and asserting to a vision-language model that reads screenshots rather than the underlying DOM. It plugs into Playwright and Puppeteer for web automation, uses adb for Android, and WebDriverAgent for iOS.

The interesting bit

The project has gone all-in on pure-vision interaction: it removed DOM-based action localization entirely, relying on screenshots alone to find buttons and fields. That means it can operate on anything renderable—including mobile apps and canvas-based UIs—while supposedly burning fewer tokens and less time than traditional selector-based approaches. DOM is still available as an optional side dish for data extraction when you need structured page context.

Key highlights

• Cross-platform reach: web (with Playwright/Puppeteer), Android (via adb), iOS (via WebDriverAgent), and generic interfaces via a JavaScript SDK.
• Natural-language scripting: describe a task in YAML or JS and let the model plan the steps.
• Built-in debugging toolkit: visual replay reports, an interactive playground, and a Chrome Extension for zero-code experimentation.
• MCP server support: exposes atomic agent actions as tools for upstream AI systems.
• Self-hosting friendly: supports open-source vision models including Qwen3-VL, UI-TARS, and Doubao-1.6-vision, plus a caching layer for faster replays.

Verdict

Worth a look if you maintain end-to-end tests across web and mobile and are tired of brittle selectors, or if you need to automate surfaces where DOM access simply does not exist. Less compelling if your stack is already deeply invested in traditional programmatic automation and you distrust LLM latency.