Open-source telemetry for LLM apps that actually runs locally

What it does

Phoenix is an open-source observability platform for LLM applications. It captures runtime traces via OpenTelemetry, benchmarks outputs with built-in evaluators, versions datasets for experiments, and manages prompts through a web UI. It runs locally, in a notebook, containerized, or in Arize’s managed cloud.

The interesting bit

Most AI observability tools are SaaS-only dashboards that charge by the span. Phoenix inverts the model: the full platform is open-source and self-hostable, with lightweight sub-packages (phoenix-otel, phoenix-client, phoenix-evals) that let you instrument code without pulling in the entire server. The breadth of auto-instrumentation is notable—it covers not just raw API calls but agent frameworks like OpenAI Agents SDK, LangGraph, CrewAI, and Mastra through the separate OpenInference project.

Key highlights

• OpenTelemetry-native tracing with out-of-the-box support for 10+ frameworks and providers
• Built-in LLM-as-judge evaluators for RAG relevance, answer quality, and retrieval
• Versioned datasets and experiment tracking for prompt/parameter iteration
• Web-based playground for replaying traced calls and comparing model outputs
• MCP server available for Cursor integration
• Modular packaging: use just the OTel wrapper or eval library if you already run the platform elsewhere

Caveats

• The README is heavy on feature listing and light on architecture or performance characteristics; expect to dig into docs for production deployment patterns
• “Runs practically anywhere” is claimed but no specific resource requirements are given for self-hosted instances

Verdict

Worth evaluating if you’re building non-trivial LLM apps and want to own your telemetry pipeline. Probably overkill if you’re just calling openai.chat.completions.create a few times a day and don’t need systematic evals or prompt versioning.