An open agent that clicks, types, and outscores humans

What it does

Agent S drives your actual mouse and keyboard through the GUI to complete tasks on Linux, macOS, or Windows. It pairs a main reasoning model with a dedicated grounding model that translates plans into exact screen coordinates, then executes actions via generated Python code. A reflection agent monitors the worker, and an optional local coding environment lets the agent drop to Python or Bash for file manipulation or data processing.

The interesting bit

The project frames computer use as an in-context reinforcement-learning problem: the agent learns from past trajectories and can sample multiple rollouts, picking the best one to boost accuracy. That “Behavior Best-of-N” trick is what pushed its OSWorld score to 72.6%, just above the roughly 72% human baseline.

Key highlights

• Surpassed human-level performance on OSWorld (72.60% with Behavior Best-of-N, 66% raw) and posts strong zero-shot results on WindowsAgentArena and AndroidWorld.
• Modular architecture: swap in different LLM providers (OpenAI, Anthropic, Gemini, etc.) and grounding models such as UI-TARS-1.5-7B.
• Includes a reflection agent to assist the worker agent and an optional local code-execution environment for non-GUI automation.
• Runs natively on the host machine with the same user permissions, meaning it can genuinely control your desktop.
• Backed by peer-reviewed papers accepted at ICLR 2025 and COLM 2025, plus a best-paper award at an ICLR workshop.

Caveats

• The optional local coding environment executes arbitrary Python and Bash with your user permissions; the README warns you to treat it like a stranger with your house keys.
• Requires a separate grounding model endpoint and careful resolution matching between the model output and your screen.
• Single-monitor support only; multi-monitor setups are explicitly out of scope.

Verdict

Worth a look if you are building or benchmarking GUI agents and need an open alternative to closed computer-use APIs. Skip it if you want a hands-off cloud service that does not require hosting a grounding model or trusting locally executed code.