IBM's factory-floor LLM benchmark: 141 scenarios for chiller whisperers

What it does

AssetOpsBench is a benchmark and scaffolding system for LLM agents that operate industrial equipment—chillers, HVAC, IoT sensor fleets, and the work orders that keep them alive. It provides 141+ reproducible scenarios across nine asset classes, plus domain-specific MCP servers that expose real-ish tool surfaces: time-series forecasting, failure-mode lookup, vibration FFT analysis, and maintenance ticket generation. The goal is to move beyond chatbot demos to agents that can reason about physical assets without tripping over their own context window.

The interesting bit

The project treats industrial maintenance as a multi-agent coordination problem, not a single prompt. It ships orchestration blueprints—MetaAgent and AgentHive—that delegate across specialist agents (IoT, TSFM, FMSR, Work Order, Vibration) over MCP, the Model Context Protocol. That means the benchmark is also a runtime: you can swap LLMs, compare trajectories, and get scored by an LLM judge on six dimensions of reasoning and execution fidelity.

Key highlights

• 141+ scenarios spanning single-domain queries and multi-step end-to-end workflows
• Native MCP support with domain-specific tool servers for sensors, forecasting, failure modes, and work orders
• Four agent frameworks included: plan-execute, deep agent with virtual filesystem, and ReAct variants for Claude and OpenAI models
• Leaderboard infrastructure using Llama-4-Maverick-17B as judge, evaluated across 7 LLMs
• Active competition track: IJCAI 2026 live challenge, plus 500+ prior submissions at CODS 2025
• 12+ accepted papers at KDD, NeurIPS, AAAI, ICLR, ACL, EMNLP, and IAAI between 2025–2026

Caveats

• The quick-start scenario runner is marked “to be enabled” in the README; the Colab and Hugging Face playground are the current entry points
• Leaderboards are listed as “WIP with latest models,” so published rankings may lag the codebase
• Active development lives on main, but publication-specific branches (e.g., ACL 2026 IndustryAssetEQA) are maintained separately, which could fragment reproducibility

Verdict

Worth a look if you’re building agents for physical infrastructure or researching multi-agent tool use in constrained domains. Skip it if you want a polished, batteries-included runtime—this is a research benchmark with sharp edges and a bibliography longer than most startup pitch decks.