A Rust LLM gateway that adds half a millisecond, not half a second

What it does

Antirion sits between your applications and every major LLM provider — OpenAI, Anthropic, Gemini, Bedrock, Mistral, Groq, DeepSeek and more, plus any OpenAI- or Anthropic-compatible server like vLLM. Clients speak whichever API they already use; Antirion translates. It also handles the boring-but-critical parts: routing with fallbacks and circuit breakers, dollar and token budgets per team or key, guardrails for PII and prompt injection, caching, and an audit log. Everything ships as one Rust binary with the admin console embedded — no sidecar, no separate UI deployment.

The interesting bit

The benchmark methodology is unusually honest: open-loop load testing on dedicated AWS c7i machines, with the load generator and upstream isolated so only the gateway is measured, and the harness published in bench/ for reproduction. The numbers are stark — 0.53 ms p50 overhead versus Bifrost’s 19.7 ms, while LiteLLM returns 100% errors at 5,000 RPS on the same box.

Key highlights

• Sub-millisecond p50 overhead and ~21,500 req/s throughput on 4 vCPUs, per the published benchmark
• 41 MiB idle memory; the whole gateway plus console fits where LiteLLM’s interpreter alone doesn’t
• Protocol translation between OpenAI, Anthropic, Gemini and Bedrock request formats
• Guardrails with format-preserving redaction, applied to requests, responses and streams
• Runs standalone with an embedded store, or as a cluster on PostgreSQL and Valkey; Helm chart included

Caveats

• The benchmarks are self-published by the project, comparing against Bifrost 2.2.4 and LiteLLM 1.103.1 — dated versions, and the authors clearly picked a fight they expected to win.
• At 500 stars, this is a young project; production claims (SSO, SCIM, clustering) are stated but not independently verified here.

Verdict

If you run meaningful LLM traffic and gateway overhead or memory footprint actually matters to you, this is worth a benchmark on your own hardware. If you’re happy with LiteLLM’s ecosystem and your traffic is modest, the status quo is less work.