An open-source traffic cop for LLM requests and budgets

What it does

LLM Gateway is middleware that parks itself between your application and various LLM providers—OpenAI, Anthropic, Google Vertex AI, and others. It offers a single OpenAI-compatible API surface so your client code doesn’t need to change, while it handles provider routing, API key management, and token-cost analytics in the background. A Next.js dashboard and playground ship alongside the Hono-based gateway for visualizing spend and performance.

The interesting bit

The real value is in the accounting layer. Most proxies can reroute an API call; this one tracks response times, token burn, and cost-effectiveness across models so you can optimize which provider actually answers your prompts.

Key highlights

• OpenAI-shaped front door: Presents a unified OpenAI-compatible API to clients, even when backend providers use different protocols.
• Spend tracking: Logs requests, tokens, response latency, and costs for cross-provider comparison.
• Batteries-included stack: Ships as a monorepo with a Next.js UI, Hono API backend, playground, and Drizzle ORM schemas.
• Dual license: Core gateway and analytics are AGPLv3; enterprise additions such as team management, billing, and extended retention are gated in an ee/ directory.

Caveats

• Data retention is split by license: the README contrasts enterprise “unlimited” retention against a 30-day window, suggesting the open-source build is capped there.
• The unified self-hosted container bundles PostgreSQL and Redis and is picky about Docker volume permissions; bind-mounting host directories directly into the Postgres path can fail.
• The enterprise feature list ends with “And more to be defined,” so the commercial roadmap is still partly a sketch.

Verdict

Good fit for small teams or startups juggling multiple LLM providers and tired of scattershot API keys and surprise bills. Large organizations needing hardened security policies or advanced prompt guardrails will likely outgrow it quickly.