Give your AI agent a pop quiz on its own blind spots

What it does iFixAi is a fixture-driven diagnostic that probes an AI agent for misalignment risks—hallucinations, prompt-injection susceptibility, oversight evasion, and more—and returns a letter-grade scorecard in minutes. It is model- and provider-agnostic, designed to drop into CI pipelines, and tracks drift by comparing repeated runs against the same fixture. The maintainers are explicit that it is not a safety certification, just a reproducible signal for whether your agent is improving or regressing.

The interesting bit The tool refuses to let a model grade its own homework. By default, the CLI demands a second provider credential and automatically uses a different judge model to score the system under test, turning evaluation into a cross-examination. Every run also writes a content-addressed manifest so the exact inputs can be audited or replayed later.

Key highlights

• 32 core inspection points across five risk pillars, plus extended checks for frontier risks such as sandbagging and oversight evasion.
• Cross-provider judging by default: the SUT is scored by a different LLM backend unless you explicitly opt into self-judge mode.
• Built for CI drift tracking and A/B comparisons on identical fixtures, with scorecards written locally after each run.
• Content-addressed manifests capture every input for reproducibility and audit trails.
• Vanilla LLMs score 27 of 32 inspections out of the box; wiring governance hooks unlocks the full scorecard.

Caveats

• No published baselines yet, and default thresholds are policy defaults—not empirically calibrated—so absolute scores are informative, not authoritative.
• The animated demo in the README showcases a custom client build; the open-source CLI behaves and presents results differently.
• Five inspections depend on governance hooks; without them, those checks are skipped or scored from inline fixture declarations with a warnings[] flag.

Verdict Worth a look if you ship AI agents and want a fast, repeatable sanity check before release. Skip it if you need a certified safety guarantee or turnkey regulatory compliance.