Your production failures, frozen into CI tests that block PRs

What it does

Tracely ingests agent traces over plain OTLP, grades every run as it lands (LLM-as-judge plus model-free structural checks), and clusters failures into issues instead of 31 rows to read. One click promotes a failing trace into a hermetic regression case — recorded inputs, tool and LLM outputs bundled as fixtures — which replays in CI against recorded data. tracely gate exits non-zero and posts a commit status, so a regression actually blocks the PR rather than nudging a dashboard number.

The interesting bit

The core bet: the recorded run is the test. Instead of guessing what might break in a hand-authored dataset, you freeze the exact failing run byte for byte, and replay costs nothing — no API keys, no model spend. Promotions also carry a fail-to-pass contract: the case must fail on old code and pass on the fix, or the promotion isn’t trusted.

Key highlights

• Agent semantics (agent.id, turn, step) become indexed columns, so runs group into conversation threads rather than flat span soup
• Failure clustering combines structural signatures with semantic embeddings, and suggests the evaluator that would have caught each cluster
• Multi-turn behaviour gets “scenarios”: scripted conversations or adversarial goals a red-team model improvises against
• Judge calibration screen labels verdicts against human review, so you catch an over-flagging judge before it gates a release
• Alert rules are visual flows with conditions, Slack/email/webhooks, an LLM step, and Python expressions; the API doubles as an MCP endpoint

Caveats

• Self-hosting the full stack means running Postgres, ClickHouse, Redis and MinIO alongside the app — a real footprint, even if one-click deploy templates exist
• The README is dense with features; how well clustering and judge suggestions work in practice isn’t demonstrated with numbers

Verdict

Teams running agents in production who are tired of maintaining eval datasets by hand should look hard at this. If you don’t have production traffic yet, there’s nothing to freeze — stick with hand-authored evals for now.