When Agents Have to Actually Run the Store

What it does RealReplicaBench, built by Alibaba International’s Accio team, tests whether agents can execute multi-step business workflows across browser, CLI, file, and API surfaces. Each of the 107 tasks spins up a fresh container against local, stateful mock replicas of real commerce and SaaS tools, then grades success with deterministic or LLM-assisted verifiers.

The interesting bit Instead of testing reasoning in the abstract, it traps agents inside high-fidelity replicas of actual software and demands state changes. The mock services model everything from logistics platforms to document systems, so an agent must navigate real UI patterns and leave auditable traces, not just emit plausible text.

Key highlights

• 107 tasks spanning CLI, browser, file, and API/MCP workflows, with 22 requiring vision
• Local mock replicas eliminate the need for production accounts while preserving realistic statefulness
• Dual harness support (OpenClaw and Accio) with published reference results for roughly a dozen model families
• Fully auditable runs: every trajectory, artifact, log, and container state is preserved
• The maintainers evaluate private and pre-release models on request

Caveats

• Raw task-level result bundles are not yet stored in Git or available as immutable public artifacts; the leaderboard is an audited aggregate rather than a fully reproducible package
• The public evaluation path requires bring-your-own credentials for both the model and the LLM judge

Verdict Worth your time if you are building or evaluating long-horizon agents that need to survive messy business UIs. Skip it if you are looking for a pure reasoning or chat-oriented benchmark.