A Petri dish for testing LLM alignment hypotheses

What it does Inspect Petri is an automated auditing agent that probes language models for misalignment. You supply seed instructions describing a hypothesis, and it generates realistic scenarios, orchestrates multi-turn dialogues between an auditor model and a target model, simulates tools and rollbacks to probe edge-case behaviors, then scores transcripts with a judge model against a consistent rubric.

The interesting bit The design treats alignment testing as a controlled experiment with separated roles: one model interrogates, another responds, external tools can be simulated or rolled back, and a distinct judge evaluates the transcript. This keeps the audit from collapsing into a single-model echo chamber.

Key highlights

• Generates realistic audit scenarios from seed instructions
• Orchestrates multi-turn audits between an auditor and target model
• Simulates tool use and rollbacks to test edge-case behaviors
• Scores transcripts with a dedicated judge model using a consistent rubric
• Version 3.0 refactors internal Python APIs; v2 remains available on a separate branch

Caveats

• The README is light on implementation specifics—model providers, supported formats, and configuration details are left to the external documentation site
• Version 3.0 broke backward compatibility with internal Python APIs, so existing integrations may need work

Verdict Worth a look if you need automated, repeatable alignment audits for language models. Skip it if you were hoping for exhaustive technical detail in the repo itself—the README sends you to external docs for the heavy lifting.