1,671 ways to test if your agent can actually practice law

What it does

Harvey LAB is a dataset plus an execution harness for grading LLM agents on real legal workflows. The dataset contains 1,671 tasks spanning more than 24 practice areas, each with instructions, source documents, and rubrics. The harness runs agents against these tasks and scores them, offering comparison dashboards and sweep support.

The interesting bit

Most legal AI benchmarks test static Q&A; this one drops agents into realistic environments with full document sets and expects them to perform. The included tutorial walks through an M&A data-room assignment end to end, which is a nice touch for a field that usually hides behind academic abstracts.

Key highlights

• 1,671 tasks across 24+ legal practice areas plus contracting, grounded in real documents and rubrics.
• Execution harness manages task inspection, agent runs, scoring, and report generation—not just a static CSV.
• Built-in support for evaluation sweeps and comparison dashboards.
• Tutorial centers on a realistic M&A data-room workflow, not toy examples.
• MIT licensed and actively maintained, with public contribution guidelines.

Caveats

• The project is explicitly ongoing, so the task set and harness are still being refined.
• The README is light on concrete performance numbers or baseline leaderboards.

Verdict

Worth a look if you’re building or evaluating legal AI agents. Skip it if you’re after a finished, static benchmark with established leaderboards.