Self-hosted tracing and judgment for agent runs

What it does PandaProbe is a full-stack observability and evaluation platform for AI agents. It ingests traces and spans from agent runs, queues them via Redis for async processing by Celery workers, and surfaces them through a Next.js dashboard backed by FastAPI. You can self-host the entire stack or fall back to the managed cloud tier.

The interesting bit The architecture splits the control plane (user auth via Supabase or Firebase) from the data plane (trace ingestion via API keys), which is a sensible posture for a tool handling potentially sensitive agent traffic. Evaluation jobs are dispatched to an LLM engine through LiteLLM, using an LLM as a judge to score agent behavior asynchronously.

Key highlights

• Ships as a complete Dockerized stack: Next.js frontend, FastAPI app, Celery workers, PostgreSQL, and Redis.
• Claims integrations with LangGraph, CrewAI, Claude Agent SDK, and OpenAI Agents SDK, though the README pushes specifics to external docs.
• Evaluation pipeline uses LLM-as-a-judge via LiteLLM, with async job processing through Redis and Celery.
• Apache 2.0 licensed, with a managed cloud option if you do not want to operate the fleet yourself.

Caveats

• The README is mostly an architecture diagram and a start script; integration details, SDK usage, and configuration depth live in external documentation.
• It is unclear from the repository alone how evaluation criteria are defined or how deeply the framework integrations instrument agent internals versus simply wrapping HTTP calls.

Verdict Worth a look if you are shipping agents to production and need a self-hosted alternative to proprietary observability platforms. Skip it if you want a lightweight, embedded SDK without the infrastructure overhead of PostgreSQL, Redis, and Celery.