OpenAI's sample app teaches agents to drive a browser — carefully

What it does

This is OpenAI’s sample app for CUA, their Computer Using Agent, wired through the Responses API. It pairs a Next.js operator console with a Fastify runner that spins up isolated workspaces and Playwright browser sessions, then lets the model either click around like a human (native mode) or write its own Playwright JavaScript (code mode). Three bundled labs — a kanban board, a paint canvas, and a booking flow — give it something deterministic to actually accomplish.

The interesting bit

The dual-mode design is the lesson: the same scenario manifests and replay pipeline work whether the model emits raw computer actions or scripts the browser through a persistent exec_js REPL. It’s a clean side-by-side of “agent as mouse-wiggler” versus “agent as programmer,” and the replay bundles mean you can watch what happened after the fact.

Key highlights

• All Responses API integration lives in one canonical file: packages/runner-core/src/responses-loop.ts
• Scenario manifests with built-in outcome verification — runs are checked against target state, not just eyeballed
• Operator console stays legible even when the runner is offline or a run fails
• The old Python sample is gone from this branch; history lives on a v1/legacy branch
• Live smoke tests are opt-in and gated behind your own API key

Caveats

• Safety acknowledgements aren’t implemented — runs fail with unsupported_safety_acknowledgement if the API requests one
• Strictly browser-focused; workspace patching and file-editing scenarios are out of scope
• The scenarios are local labs for deterministic verification, not evidence of general web autonomy — OpenAI says so themselves

Verdict

Worth a read if you’re integrating computer-use agents and want a well-structured reference for the Responses loop, replay, and verification. If you wanted a general web-autonomy demo, this isn’t that — and it’s honest about it.