One Rust binary to translate every AI API into every other

What it does It’s a single Rust binary that sits between your code and every AI provider. Point the OpenAI, Anthropic or Gemini SDK at it and it translates formats on the fly — an Anthropic client can call an OpenAI model, an OpenAI Realtime client can talk to Gemini Live. It also handles images, speech, transcription, embeddings and video, plus traffic control: named routes, fallbacks, retries, and hedged requests that race a slow provider against a second one.

The interesting bit MCP servers and built-in tool vendors (web search, code sandboxes) sit behind the gateway and run inside ordinary chat requests, so any client/model combo gets tools without knowing about them. And every feature is a plugin on a public Rust API — one trait, four hooks — so guardrails or billing are extension points, not forks.

Key highlights

• Zero-config start: any vendor key in the environment becomes a provider; no config file needed
• Claims 0.07 ms added latency, ~56k req/s and 31 MB memory in their own benchmarks
• Live request log with cost, latency and tokens per call; optional tracing in Respan’s platform
• Runs as a server, in-process inside Python, or embedded in your own Rust service
• Refuses to start on a public address without client keys — a sensible default

Caveats

• The performance table is the project’s own benchmark; the methodology isn’t shown in the README
• Observability beyond the local log ties into Respan’s commercial product

Verdict If you’re juggling multiple providers and want one self-hosted, fast, polyglot endpoint — this is a serious LiteLLM alternative. If you’re happy with a single vendor, it’s extra machinery you don’t need.