← all repositories
sunblaze-ucb/exploitgym

869 real bugs, one question: can AI agents actually exploit them?

Built from real userspace, V8, and Linux kernel vulnerabilities, ExploitGym exists to answer one uncomfortable question: can AI agents turn bugs into working exploits?

exploitgym
Not currently ranked.
star history

What it does ExploitGym packages 869 tasks (v1.0) built from real-world vulnerabilities in userspace programs, Google’s V8 engine, and the Linux kernel, and scores AI agents on whether they can develop working exploits. Targets run as Docker images the agent must actually compromise; success is checked against flags derived at runtime, not stored anywhere in the repo. The companion paper’s title states the agenda without euphemism: “Can AI Agents Turn Security Vulnerabilities into Real Attacks?”

The interesting bit The plumbing is the point. The controller generates its token salt, flag seed, and API key at startup, and both ends must share the same values — so expected flags can’t be scraped off GitHub. Agent containers sit behind a Squid firewall for outbound network isolation, so a model can’t phone home or fetch the answer. This is benchmark design for a contestant that cheats by default.

Key highlights

  • 869 instances across three target classes: userspace programs, V8, and the Linux kernel
  • Real-world vulnerabilities rather than synthetic exercises
  • Runtime-derived flags and per-run secrets — nothing exploitable in the repo itself
  • System defenses like ASLR can be disabled, per the docs, separating raw exploitation skill from defense bypass
  • Author list spans Carlini, Song, Holz, and Shoshitaishvili; source is Apache-2.0, task data keeps its upstream licenses

Caveats

  • Operationally heavy: per-task-family Docker images, GDB, socat, node, agent CLIs, and three long-running services (controller, firewall, LLM proxy). A lab instrument, not a lightweight harness.
  • It’s an LLM-agent evaluation by design — model API keys and the proxy sit in the loop.
  • The README reports no scores; for how agents actually fare, read the paper.

Verdict Care if you research AI offensive capability, agent evaluation, or benchmark integrity under adversarial pressure. Skip it if you wanted a casual CTF sandbox — the operational weight is real, and the point is measurement, not practice.

Frequently asked

What is sunblaze-ucb/exploitgym?
Built from real userspace, V8, and Linux kernel vulnerabilities, ExploitGym exists to answer one uncomfortable question: can AI agents turn bugs into working exploits?
Is exploitgym open source?
Yes — sunblaze-ucb/exploitgym is open source, released under the Apache-2.0 license.
What language is exploitgym written in?
sunblaze-ucb/exploitgym is primarily written in C.
How popular is exploitgym?
sunblaze-ucb/exploitgym has 1.3k stars on GitHub.
Where can I find exploitgym?
sunblaze-ucb/exploitgym is on GitHub at https://github.com/sunblaze-ucb/exploitgym.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.