Treat AI red-teaming as scenario engineering, not prompt tricks

What it does

Promptbeat is a safety evaluation toolkit that generates adversarial test cases against live LLMs and agent applications. It combines target profiles, risk scenarios, and seed material into structured attack campaigns, then evaluates the results against real models or agents. This public repository holds documentation, runnable configuration examples, and dataset subscription templates, but explicitly excludes the core product source and raw benchmark datasets.

The interesting bit

The project treats red-teaming as a systems problem rather than a creative writing task. Instead of hand-crafting individual jailbreak prompts, you define a scenario with success criteria and judge strategy, point it at a target via an adapter, and let the framework generate and evaluate the attack surface. That abstraction holds whether the target is a raw LLM API, an HTTP agent endpoint, or a Codex SDK coding runtime.

Key highlights

• Scenario-driven testing that separates risk taxonomy, success criteria, and judge strategy from raw prompt generation.
• Dataset subscriptions that reference catalogs like HarmBench, JailbreakBench, Do-Not-Answer, and XSTest when raw files are available locally.
• Adapter-based targeting for HTTP agents, Codex SDK coding agents, and templates for Claude Code, OpenCode, and OpenClaw.
• Bilingual documentation site (English and Chinese) maintained independently in website/.
• Public repository contains runnable examples and subscription templates, though the core engine and API service remain closed-source.

Caveats

• The core product source, API service, and raw benchmark datasets are not included here; this is a docs-and-configs repository.
• Several agent adapters (Claude Code, OpenCode, OpenClaw) are currently unvalidated templates awaiting connection to real runtimes and saved reports.
• Dataset subscriptions require locally supplied raw files; they are not bundled in the release.

Verdict

Useful if you need a structured, repeatable framework for red-teaming production LLMs or agent systems. Not the right choice if you are looking for a fully open-source security testing engine or turnkey benchmark datasets.