A test harness that treats agents like software under test

What it does Harbor is a Python framework for running agent evaluations and building RL environments. It wraps third-party benchmarks (SWE-Bench, Aider Polyglot, Terminal-Bench) and arbitrary agents (Claude Code, OpenHands, Codex CLI) in Docker containers, then dispatches them locally or to cloud providers like Daytona and Modal. Results feed back into RL training pipelines as rollout data.

The interesting bit The framework decouples agents, models, datasets, and execution environments so you can mix and match. Want to test Claude Code against Terminal-Bench 2.0 on 100 Daytona instances? One flag change. The same abstraction lets you swap in your own benchmark or agent without rewriting orchestration logic.

Key highlights

• Ships with integrations for major coding agents and benchmarks; harbor datasets list shows the full catalog
• Scales from local Docker (--n-concurrent 4) to cloud execution (--n-concurrent 100 --env daytona) via CLI flags
• Generates rollouts formatted for RL optimization workflows
• Official harness for Terminal-Bench 2.0, which originated the project
• Companion cookbook repo hosts end-to-end examples

Caveats

• README is thin on internals: unclear how environments are isolated, how results are scored, or what the RL data format looks like
• Cloud provider integrations require separate API keys and presumably cost money; no pricing guidance given

Verdict Worth a look if you’re benchmarking coding agents or building RL training pipelines around tool-use LLMs. Skip it if you just need a one-off eval script — the overhead only pays off at scale or with repeated experiments.