Self-hosted AI red teamers that sandbox their own exploits

What it does

PentAGI is a self-hosted, microservices-based platform that orchestrates multiple LLM-powered agents to perform penetration tests. Each agent gets a sandboxed Docker environment stocked with actual security tools—nmap, Metasploit, sqlmap, and roughly twenty others—and can search the web, query a knowledge graph, or consult long-term memory stored in PostgreSQL with pgvector. The system exposes both REST and GraphQL APIs and feeds telemetry into Grafana, Jaeger, and Langfuse so you can watch the machines think in real time.

The interesting bit

The architecture reads like a full observability stack that happens to do offensive security: Neo4j for a knowledge graph, VictoriaMetrics and Jaeger for telemetry, ClickHouse for analytics, plus a custom web scraper and seven different search integrations. The unusual part is the explicit “Current Capability Boundaries” section, which admits the system is not a CALDERA-style adversary-emulation product and that fully autonomous agent-authored attack scripts remain future work rather than current reality.

Key highlights

• Sandboxed execution: every agent action runs inside an isolated Docker container, not on your laptop.
• Multi-agent delegation: distinct agent types (researcher, developer, executor) handle different phases of a test.
• Broad LLM support: works with OpenAI, Anthropic, Ollama, AWS Bedrock, DeepSeek, GLM, Kimi, Qwen, and custom OpenAI-compatible endpoints; includes a guide for local vLLM deployments.
• Deep observability: ships with Grafana, Langfuse, Jaeger, Loki, and VictoriaMetrics pre-integrated.
• Knowledge graph memory: uses Graphiti and Neo4j to track semantic relationships across test flows, alongside pgvector for vector search.

Caveats

• The “fully autonomous” branding is slightly undercut by the maintainers’ own admission that the platform is currently assistant-guided and that true autonomous attack-script generation is not yet implemented.
• JSON export for flow reports is not documented as a supported output format today; reports are limited to web view, clipboard, Markdown, and PDF.

Verdict

Security teams with self-hosted infrastructure and a high tolerance for observability dashboards should take a look; if you need a mature, predefined adversary-emulation framework like CALDERA, this is explicitly not it yet.