Audit your AI middleman before it audits you

What it does

API Relay Audit is a local security scanner for third-party AI API relays and LLM proxies. You point it at a relay URL with your API key; it runs 14 structured checks and spits out a Markdown report with a LOW / MEDIUM / HIGH verdict. No pip install, no web dashboard holding your credentials—just curl a single Python file and run it.

The interesting bit

The tool treats “inconclusive” as a first-class citizen. Blocked probes and ambiguous responses don’t get quietly marked clean; they stay visible in the report. That’s a deliberate design choice in a space where many scanners would rather give you a green checkmark than admit uncertainty.

Key highlights

• Zero dependencies: standalone audit.py uses only Python stdlib plus curl
• Detects prompt injection, model substitution, tool-call rewriting, SSE stream anomalies, and error-response leakage
• Web3 profile adds wallet-safety probes: ETH transfer guidance, signed-transaction refusal, private-key refusal
• Modular dev version (api_relay_audit/ package) with tests for contributors
• Agent skill integrations for OpenClaw and Hermes workflows
• AGPL-3.0 license, with explicit intent to keep modified network-service deployments accountable

Caveats

• The tool does not certify safety; it produces evidence for human review
• Some probes are model-agnostic by necessity, which may limit specificity for niche model behaviors
• Web3 checks are profile-gated and may not cover all wallet interaction patterns

Verdict

Worth a run if you buy API access through resellers, use Claude/OpenAI-compatible proxies, or let agent workflows touch package installs or wallet operations. Skip it if you only hit first-party APIs directly—though reading the fixture report is still decent threat-modeling homework.