A benchmark that forces coding agents to finish what they start

What it does

DeepSWE is a 113-task benchmark that pits coding agents against software engineering work drawn from active TypeScript, Go, Python, JavaScript, and Rust repositories. Each task runs in an isolated environment where the agent must commit its work; a separate verifier later applies the resulting patch in a pristine container and grades only the observable behavior, not internal structure. The reference solution is kept offline and never used during scoring.

The interesting bit

The setup deliberately separates the agent’s sandbox from the grading environment, so an agent can’t game the verifier by peeking at tests or leaking state. The companion runner, Pier, fine-tunes network access per agent—giving it only the endpoints it needs while keeping the task environment isolated—which sidesteps the usual air-gapped-evaluation headache where dependency installs and LLM calls get blocked.

Key highlights

• 113 tasks across five languages, all sourced from active open-source repos rather than synthetic puzzles.
• Grading is behavior-based: the verifier checks observable output, so renaming variables or refactoring internals won’t tank the score.
• The agent environment and verifier are strictly separated; the agent commits work, and grading happens in a fresh container from that patch.
• The benchmark ships in the Harbor task format and is designed to run via Pier, which adds per-agent network allowlists and richer trajectory metadata.
• Pier can drive multiple agents—mini-swe-agent, claude-code, codex, gemini-cli, and opencode—and runs parallel sandboxes on Modal.

Caveats

• Since v1.1, the benchmark requires Pier ≥0.3.0 and the separate-verifier workflow; older Harbor-only setups are unsupported.

Verdict

Worth a look if you are building or evaluating coding agents and need a realistic, tamper-resistant benchmark. Skip it if you just want a quick script to test a local LLM on LeetCode-style snippets; this is built for long-horizon, multi-file engineering.