Trust, but cryptographically verify your AI API relay

What it does

You hand it a base_url + api_key + model and it fires a battery of probe requests, then compares the responses against official-API baselines at the field, protocol, and cryptographic level. It answers three questions: is the relay actually forwarding to the model it claims, are advanced features (PDF, tool use, thinking, function calling, long context) silently stripped, and do response fields, streaming sequences, and usage accounting match the official spec. It covers the Anthropic Messages API, OpenAI Chat Completions, and Gemini’s OpenAI-compatible API, and is self-hostable (Python, FastAPI) or usable via the free hosted service.

The interesting bit

The cleverest check is Claude’s thinking_signature: when thinking is enabled, Anthropic’s server returns a cryptographically signed signature field (~500–2000 chars) that a relay theoretically cannot forge — the one check that yields a near-mathematical authenticity verdict, weighted at 25% for the Claude protocol. For OpenAI/Gemini, which lack server-side signatures, it falls back to behavioral and protocol-level checks, including spotting “re-chipped” relays via leftover fingerprints like claude_cache_creation_* in the usage field.

Key highlights

• 12 detectors for Claude, 8 for OpenAI, 7 for Gemini, organized into authenticity / capability / compliance categories
• Three-tier needle-in-haystack long-context probes catch relays advertising 1M context but actually serving 200k
• Reports come with permanent share links and JPG cards for posting to forums and group chats
• Community leaderboard aggregates 44,000+ public detections across 6,500+ relay providers, Bayesian-weighted
• Three run modes (quick/standard/full) trade cost vs. depth; full detection on Haiku runs about $0.02 on your own key

Caveats

• Probe costs are paid by your API key — extreme long-context tests can run up to ~$8 depending on the model
• Long-context probing doesn’t support Gemini yet
• The hosted service’s traffic and detection counts are self-reported in the README, not independently verifiable

Verdict

Anyone buying access through Chinese-market-style API relays — or running one and wanting to prove their own honesty — should care a lot. If you only ever hit official endpoints directly, there’s nothing here for you.