A 122 KB traffic controller for 1,600 LLMs

What it does

Portkey Gateway is an OpenAI-compatible proxy that sits between your application and a sprawling list of AI providers. It normalizes requests so you can call language, vision, audio, and image models through one API, while a bundled MCP Gateway adds authentication and observability over Model Context Protocol servers.

The interesting bit

The open-source build claims a 122 KB footprint and sub-millisecond latency, yet it packs features like SOC2 compliance, smart caching, and role-based access control. Portkey is also merging its commercial core back into the open-source repo for a 2.0 release, which either signals genuine openness or a very patient upsell strategy.

Key highlights

• Routes to 1,600+ models across 250+ providers using a single, familiar API signature
• Built-in reliability: automatic retries with exponential backoff, fallbacks on specified errors, load balancing, and request timeouts
• 40+ pre-built guardrails for input/output filtering, plus support for custom and partner guardrails
• Multi-modal support covering vision, audio (TTS/STT), image generation, and real-time WebSocket APIs
• Smart caching to cut costs and latency

Caveats

• The README cites “<1 ms latency” and “10B tokens processed everyday” without benchmarks or methodology, so treat those figures as unverified marketing metrics.
• Enterprise-only features such as org management and advanced governance are documented separately; the exact feature split between open source and paid tiers is not clearly itemized.

Verdict

Worth a look if you run production traffic across multiple AI providers and want guardrails, caching, and failover in one layer. Overkill if you only ever call a single model and don’t need request orchestration or policy enforcement.