Show, don't tell: teaching GUI agents by demonstration

What it does

OpenAdapt is an open-source pipeline for automating desktop and web GUI work — think EHRs, payer portals, and other software nobody enjoys clicking through. You record a human demonstration, the platform scrubs PII/PHI from it, and then either retrieves similar demos at runtime or fine-tunes a vision-language model on them. The result is an agent that executes the task with a safety gate validating each action before it fires.

The interesting bit

The core thesis is that zero-shot VLMs fail at GUI tasks not because they’re dumb, but because UI affordances are ambiguous — three buttons look like “submit.” Conditioning the agent on a retrieved human trajectory fixes that: on a 45-task macOS System Settings benchmark, first-action accuracy went from 46.7% to 100%, with a length-matched control ruling out the “it just got more tokens” explanation. The claimed differentiator is demo-conditioned inference — OpenCUA reused their capture code but only trains on demos, never conditions on them at runtime.

Key highlights

• Three-phase pipeline: demonstrate, learn, execute — with a unified CLI over a modular meta-package architecture
• Policy/Grounding separation: one component decides what to do, another maps it to where on screen
• PII/PHI scrubbing built in via openadapt-privacy, which matters a lot for the healthcare use case
• Safety gate with confirm mode for high-risk actions before execution
• Works with Claude, GPT-4o, and Qwen3-VL; fine-tuning path supports models like qwen3vl-2b

Caveats

• The headline 100% number is first-action accuracy on 45 tasks sharing a common entry point — a narrow benchmark, and the README points to a publication roadmap for limitations
• Demo-conditioned fine-tuning (the “unique value” cell) is still in progress; only retrieval-based prompting is validated
• The 100%-accuracy demo-conditioned result and the “no model calls, deterministic replay” framing in the repo description describe different execution modes; the README doesn’t fully reconcile them

Verdict

Worth a look if you’re building computer-use agents or automating click-heavy legacy apps and want a demo-conditioned approach with privacy tooling included. If you wanted a finished, fine-tuned production model today, you’re getting a validated research direction plus a solid capture stack instead.