A Red-Team Benchmark Suite, Retired by Progress

What it does

This repository houses 104 Jeopardy-style Capture The Flag challenges built in Docker, each hiding a flag that an AI or human must extract through web exploitation. Created by XBOW Engineering and external contractors, the benchmarks mirror real-world vulnerability classes encountered in pentesting and bug bounty work.

The interesting bit

The suite was kept confidential until release to avoid training-data contamination, and includes Alignment Research Center canary strings explicitly warning against ingesting the data. Despite that precaution, the maintainers note that by mid-2026 the underlying vulnerabilities had been “trained into the models” through general improvement, pushing industry-standard performance to approximately 100% and forcing them to retire their own benchmark suite as a historical artifact.

Key highlights

• 104 standalone CTF challenges with configurable flag injection via Docker build args
• Three difficulty tiers (easy, medium, hard) with metadata tags and win conditions
• Explicit canary strings to detect if the benchmarks appear in LLM training corpora
• Apache 2.0 licensed, though every benchmark is intentionally riddled with unpatched vulnerabilities

Caveats

• The maintainers explicitly warn these benchmarks are “saturated” and no longer useful for discriminating between models or attack frameworks
• Every benchmark contains deliberate, unfixed security flaws; running them outside an isolated sandbox is asking for trouble
• The repository is preserved for historical purposes only

Verdict

Grab this if you are studying the brief shelf life of AI security evaluations or need a template for Docker-based CTF infrastructure. Skip it if you are looking for a current, meaningful way to stress-test a modern red-team LLM.