The AI intern who actually answers pagers at 3am

What it does

HolmesGPT is a CNCF sandbox project that runs an agentic loop across your observability stack to find root causes during incidents. It queries live data from Prometheus, Kubernetes, cloud APIs, databases, and SaaS tools, then reasons through the results. A new “Operator mode” runs 24/7 in Kubernetes, spots problems automatically, and can message Slack or even open GitHub PRs with fixes.

The interesting bit

The project treats LLM context windows as a scarce resource worth defending. It does server-side filtering, JSON tree traversal, and output transformation before data ever reaches the model, plus per-tool memory limits and disk streaming for large observability payloads. This is the rare AI tool whose README admits petabyte-scale data exists and plans accordingly.

Key highlights

• 40+ built-in data source integrations including Prometheus, Datadog, Grafana, AWS, Azure, GCP, and various databases
• Bidirectional alert routing: ingests from PagerDuty, OpsGenie, AlertManager, Jira; writes findings back
• Pluggable LLM support: OpenAI, Anthropic, Azure, Bedrock, Gemini, and others
• Kubernetes Remediation MCP toolset can apply fixes like scaling or rollbacks
• Originally from Robusta.dev; Microsoft is listed as a major contributor

Caveats

• Operator mode itself requires Kubernetes even though health checks can target non-K8s infrastructure
• The README’s integration table is extensive but uneven: some tools are native, others labeled “(MCP)” without explaining what that distinction means for capabilities or setup

Verdict

Worth evaluating if your on-call rotation is tired of being human glue between Slack, Grafana, and kubectl. Less compelling if your incidents are already well-instrumented with low-cardinality metrics and fast runbooks — though the automated PR opening might still raise eyebrows.