The open-source control panel for LLM chaos

What it does Langfuse is an open-source platform for the unglamorous parts of shipping AI apps: tracing chains and agent runs, versioning prompts, collecting evaluations, and debugging sessions. It can be self-hosted or used as a managed cloud service. The project is built on ClickHouse and comes from YC W23.

The interesting bit Most observability tools treat LLMs like slow databases. Langfuse leans into the weirdness: it traces retrieval steps, embedding calls, and multi-turn agent actions as first-class citizens, not just wrapper spans. The prompt-management layer includes server- and client-side caching so iterating on copy doesn’t add latency to production.

Key highlights

• Drop-in instrumentation for OpenAI SDK, LangChain, LlamaIndex, LiteLLM, Vercel AI SDK, and others
• Evaluation support spans LLM-as-a-judge, code evaluators, user feedback, manual labeling, and custom pipelines via API
• Datasets and benchmarks for pre-deployment testing, with hooks into LangChain and LlamaIndex
• LLM Playground lets you jump from a bad trace directly into prompt iteration
• Typed SDKs for Python and JS/TS, plus an OpenAPI spec and Postman collection

Caveats

• The README is heavy on feature lists and light on architecture or performance characteristics
• Self-hosting options are documented, but operational details (scaling, retention, cost) are left to external docs

Verdict Worth a look if your team is currently duct-taping prompts in Google Docs and debugging production via console.log. Probably overkill for a single prototype, but the self-host path keeps it honest for teams with data-residency requirements.