Your Playwright tests now require three AI opinions

What it does

Passmark is a Playwright layer that lets you describe browser actions and assertions in plain English. It feeds those descriptions to LLMs—primarily Claude and Gemini—which drive the browser via ARIA snapshots or OpenAI’s computer-use agent. A Redis-backed cache tries to skip the AI call entirely on repeat runs, and an auto-healer falls back to live AI execution when the cached path breaks.

The interesting bit

The project treats test flakiness as a consensus problem: for assertions, it runs Claude and Gemini in parallel and brings in a third arbiter if they disagree. You can also mix cheap, cacheable snapshot steps with expensive visual CUA steps in a single test run, and even evaluate assertions against full screencasts uploaded to Gemini for ephemeral UI like toasts.

Key highlights

• Multi-model assertion engine that arbitrates disagreements between Claude and Gemini.
• Per-step AI mode overrides: mix ARIA snapshot steps with OpenAI CUA visual steps in one flow.
• Redis step caching with automatic AI fallback when cached selectors or coordinates stale out.
• Video assertions via Playwright screencasts sent to Gemini’s Files API for transient UI.
• Eight configurable model slots for execution, assertions, extraction, and arbitration.

Caveats

• Requires API keys from multiple providers (Anthropic and Google at minimum; OpenAI for CUA mode), and video assertions always bypass gateways to hit Gemini directly.
• CUA mode is locked to OpenAI’s gpt-5.5 and direct API access only; no gateway support and Redis caching is skipped.

Verdict

Worth a look if your team is drowning in brittle selector maintenance and has budget for multiple LLM APIs. Skip it if you want to avoid cloud API dependencies or prefer tests without AI mediation.