Your LLM claims it can use a computer? Prove it.

What it does

AgentBench evaluates whether large language models can operate as autonomous agents across eight distinct environments. The tasks force multi-turn reasoning about operating systems, SQL databases, knowledge graphs, web shopping, and even digital card games rather than simple text completion. A newer function-calling revision, AgentBench FC, repackages five of those environments into fully containerized deployments and swaps in function-calling prompts via the AgentRL framework.

The interesting bit

It is one of the earlier academic benchmarks—published at ICLR 2024—to treat “LLM-as-agent” as a broad, cross-domain capability rather than a narrow single-task trick. The suite combines freshly built domains such as lateral-thinking puzzles with repurposed public datasets including ALFWorld and Mind2Web, so it covers both literal digital housekeeping and open-ended reasoning.

Key highlights

• Eight environments spanning OS interaction, database queries, knowledge graphs, digital card games, household tasks, web shopping, and web browsing
• AgentBench FC revision adds function-calling prompts and containerized deployment for five core tasks
• Docker Compose orchestration for the containerized task workers
• Public leaderboard with test-set scores for various models
• Backed by an ICLR 2024 paper

Caveats

• The webshop environment requires roughly 16 GB of RAM to start
• The alfworld task leaks memory and disk space until its worker process is restarted
• The knowledge-graph task depends on an online SPARQL endpoint that the authors note is currently unstable

Verdict

Reach for this if you are doing rigorous research on LLM agent capabilities and need a multi-domain stress test. Look elsewhere if you want a lightweight evaluation script you can run on a laptop between meetings.