A two-tier Envoy for the messy world of LLM APIs

What it does

Envoy AI Gateway uses Envoy Gateway to sit between application clients and generative AI services. It employs a two-tier architecture: a centralized Tier One gateway that handles authentication, top-level routing, and global rate limiting, and a Tier Two gateway that manages ingress to self-hosted model clusters with fine-grained access controls. The project integrates with a wide roster of commercial providers—OpenAI, Anthropic, AWS Bedrock, and roughly a dozen others—while also offering endpoint picking for self-hosted inference optimization.

The interesting bit

The two-tier pattern is the architectural hook. Most gateways treat all backends as a flat pool; this one explicitly separates the public API front door from the internal model cluster door, suggesting that routing to a GPU cluster deserves different guardrails and LLM-specific optimization than routing to a SaaS endpoint.

Key highlights

• Supports 16 distinct AI providers out of the box, from OpenAI and Azure to Groq, DeepSeek, and SambaNova.
• Tier One gateway centralizes auth and global rate limiting; Tier Two gateway targets self-hosted models with fine-grained access control.
• Includes an endpoint picker specifically tuned for LLM inference optimization on self-hosted clusters.
• Backed by the Envoy community and inspired by a CNCF cloud-native LLM gateway proposal.

Verdict

Worth evaluating if you already run Envoy Gateway and need to normalize access to a mix of SaaS LLMs and in-house model serving. If you are not already invested in the Envoy ecosystem, this may be more machinery than you need.