The eval step everyone skips, handed off to your coding agent

What it does It’s a pack of agent skills — markdown instruction files that a coding agent loads as domain expertise — for building product-specific evals (explicitly not foundation-model benchmarks). An entry skill, evals-start, looks at your situation and routes you: an existing eval pipeline goes to eval-audit, a pile of unanalyzed traces goes to error-discovery. From there the skills cover the rest of the workflow: synthetic test data, code checks for objective failure modes, judge prompts for subjective ones, judge calibration, RAG evaluation, and annotation interfaces.

The interesting bit The authors call error-discovery the most important skill, and it’s the most concrete: point your agent at a JSONL/CSV/JSON file of traces and it builds a single-file HTML review app served by a Python stdlib server — zero dependencies — then clusters the data, picks a diverse initial sample (cluster reps plus random picks), and loops. In the loop it watches your free-text annotations, sorts them into failure modes, and proposes new samples to fill coverage gaps. It even applies Gestalt principles to the review UI — color for categories, spacing for hierarchy, opacity for importance — which is a surprisingly design-literate instruction set for a machine that builds throwaway tooling.

Key highlights

• eval-audit inspects an existing eval pipeline and surfaces problems ranked by severity
• validate-evaluator calibrates LLM judges against human labels — data splits, TPR/TNR, bias correction — the step most judge setups quietly omit
• generate-synthetic-data creates diverse test inputs via dimension-based tuple generation
• evaluate-rag and build-review-interface cover RAG pipelines and custom annotation tooling
• The methodology is distilled from the authors’ consulting work with 50+ companies and teaching thousands of students

Caveats

• What ships is instructions, not software — the review app and tooling are generated by your agent at runtime, so output quality rides on the agent you hand it to
• Scope ends at eval work that generalizes across projects; production monitoring, regression suites, and cost live in the authors’ separate course
• The stance is uncompromising: no writing evals until you’ve done error discovery on real traces — teams wanting to jump straight to a judge score will be redirected

Verdict If your LLM product’s eval story is currently a vibe check, this is a cheap way to make your agent do the unglamorous qualitative work first. Teams with mature pipelines and actual human labelers will find it more methodology refresher than tooling.