LLM observability with Sentry-style issue tracking

What it does

Latitude ingests traces from LLM and agent executions—multi-turn sessions, tool calls, and all—and surfaces failures as grouped issues with status and trend tracking. It also generates evaluations from your team’s manual judgments, scoring how well automated evals align with human calls over time. A semantic search layer lets you retrieve traces by meaning without sampling.

The interesting bit

Instead of drowning in raw logs, you get an issue inbox where failed traces cluster together so you can track bug-like tickets for your agent. The alignment-score metric for evals is a nice touch—it catches when your automated tests drift from what humans actually think is a good response.

Key highlights

• Groups failed traces into issues with status, size, and trend tracking.
• Generates human-aligned evals from team judgments and tracks alignment drift over time.
• Captures agent-native traces: multi-turn sessions, tool calls, and full execution paths.
• Semantic search across 100% of traces by meaning, exact match, or similarity.
• Provider-agnostic telemetry via OpenTelemetry; supports OpenAI, Anthropic, LangChain, Vercel AI SDK, and OTLP backends.

Caveats

• The README funnels users toward a hosted account at latitude.so; self-hosting instructions are sparse in the docs provided.
• A stable v2 release is still pending according to the roadmap.

Verdict

Worth a look if you run AI agents in production and want crash-report-style triage rather than scrolling through raw trace logs. Less useful if you just need simple LLM cost tracking or don’t want another external dependency.