Your Agent's Last Meltdown Is Its Next Regression Test

What it does

Tracely ingests OpenTelemetry traces from AI agents and grades every run with evaluators. When it spots failures, it clusters them into issues and lets you promote a failing trace into a regression case. That case replays in CI against recorded fixtures — no live model calls, no API spend — and blocks the PR if the failure returns.

The interesting bit

Instead of asking developers to hand-author evaluation datasets that guess at what might break, Tracely treats the recorded production run as the test itself. The promotion includes a fail-to-pass contract: the case must fail on the old code and pass on the fix, or Tracely doesn’t trust it. Everything — scores, clusters, gates, trends — is derived from the trace.

Key highlights

• Trace-native observability: agent semantics like conversation.id, turn, and step are first-class indexed columns, not flat span soup.
• Zero-cost CI replay: recorded tool and LLM fixtures make regression runs fully offline and deterministic.
• Auto-clustering: structurally and semantically groups failures so 31 broken runs become one issue with a count.
• Full-stack self-hosting: ships as API, worker, UI, Postgres, ClickHouse, Redis, and MinIO — with a one-click Railway template.
• Polyglot gating: an HTTP endpoint mode lets non-Python agents gate PRs without installing a shim.

Caveats

• Operating the full stack means running Postgres, ClickHouse, Redis, MinIO, and Celery workers — not a lightweight sidecar.
• The README is bullish on the workflow but doesn’t quantify detection accuracy or clustering false-positive rates.

Verdict

Worth a look if you run non-trivial AI agents in production and are tired of maintaining synthetic eval datasets. If you don’t want to self-host a small data center, wait for a managed offering or pass.