Your GUI agent is only as good as its training trajectories

What it does ScaleCUA is a fully open-source computer-use agent and, more importantly, the large-scale dataset used to build it. The project targets the real bottleneck in GUI automation: unlike image-text pairs, operation trajectories are scarce and expensive to collect. It bundles trained models, a supervised fine-tuning codebase for Qwen2.5-VL and InternVL, an interactive playground, and an online evaluation suite spanning desktop, mobile, and web environments.

The interesting bit The work scales through a closed-loop pipeline that mixes automated agents with human experts to generate training data across six operating systems. The resulting models run behind an OpenAI-compatible API via vLLM and support two inference modes: a single end-to-end “Native Agentic Model” or a split “Agentic Workflow” that decouples UI planning from visual grounding.

Key highlights

• Reports state-of-the-art numbers on several benchmarks: 94.4% on MMBench-GUI L1-Hard, 60.6% on OSWorld-G, and 47.4% on WebArena-Lite-v2.
• Dataset covers 6 operating systems and 3 task domains, collected via automated pipelines refined by human experts.
• Training and inference supported on Qwen2.5-VL and InternVL.
• Evaluation harnesses include AndroidWorld, OSWorld, WindowsAgentArena, WebArenaLite-v2, and MacOSArena.
• Licensed under Apache 2.0.

Caveats

• The HuggingFace demo was listed as “under construction” as of the last README update.

Verdict A solid research stack for anyone training or benchmarking cross-platform GUI agents. Skip it if you want a polished consumer product instead of research infrastructure with VMs and benchmark harnesses.