An open-source vision agent that reasons before it clicks

What it does

UI-TARS-1.5 is a vision-language model that automates GUIs by ingesting screenshots and emitting structured actions—clicks, drags, scrolls, and keystrokes. It ships with three prompt templates tailored for desktop, mobile, and lightweight grounding tasks, plus a Python parser that turns the model’s text responses into executable PyAutoGUI commands.

The interesting bit

Unlike brittle script-based automation, the model uses reinforcement learning to generate explicit reasoning (“Thought”) before each action, a chain-of-thought style approach the authors credit for strong inference-time scaling. It also publishes direct benchmark comparisons against OpenAI CUA and Claude 3.7 across desktop, browser, and mobile suites, which is rare for an open-source release.

Key highlights

• Benchmark tables claim top scores on OSworld, Windows Agent Arena, Online-Mind2web, Android World, and both ScreenSpot variants versus prior SOTA and proprietary models.
• Three environment-specific prompt templates: COMPUTER_USE for desktops, MOBILE_USE for Android emulators, and GROUNDING for bare action output without reasoning chains.
• The ui-tars Python package handles post-processing, including coordinate normalization for vision models like Qwen 2.5VL.
• Adjacent projects provide a local desktop runtime (UI-TARS-desktop) and browser automation via Midscene.js.
• UI-TARS-2 expands the same architecture to games, code generation, and tool use, though details live in a separate technical report.

Caveats

• The Poki Game table shows a perfect 100.00 score on nearly every title, an eyebrow-raising result the README does not contextualize with methodology.
• Coordinate grounding is sensitive to the underlying vision model and screen resolution; the README flags that absolute coordinate handling requires careful post-processing.
• The model scale comparison table is truncated in the provided source, leaving the performance curve across smaller checkpoints unclear.

Verdict

A solid candidate if you are researching multimodal agents, building RPA alternatives, or need an open baseline to compare against proprietary automation APIs. Less useful if you want a polished end-user application; this repo is primarily the model, inference scripts, and action parsers.