The Rust load balancer that remembers what your GPUs already know

What it does

SMG is a Rust-based gateway that sits between your applications and a mix of self-hosted inference engines (vLLM, SGLang, TensorRT-LLM, MLX) and cloud providers (OpenAI, Anthropic, Gemini, xAI, and others). It exposes a single OpenAI-compatible API surface while handling routing, worker lifecycle management, multi-tenant auth, chat history storage, and MCP tool execution. Essentially, it treats HTTP and gRPC backends as interchangeable plugs behind one endpoint.

The interesting bit

The standout trick is cache-aware routing: SMG tracks each worker’s KV-cache state in radix trees to reuse prefixes across compatible engines, modeling load by looking at queued token work and KV pressure rather than just request counts. It also disaggregates prefill, decode, and even vision encoding into separate pipeline stages with DP-aware routing for data-parallel setups.

Key highlights

• Speaks OpenAI Chat Completions, Embeddings, Anthropic Messages, Gemini Interactions, the Responses API, Realtime over WebSocket/WebRTC, and MCP tool execution with approval policies
• 10 routing policies including cache_aware, least_load, and power_of_two, plus native Kubernetes pod discovery with label selectors
• Streaming gRPC pipelines to engines with prefill/decode/encode disaggregation and zero-copy tokenization caching
• Enterprise controls: per-tenant priority admission with preemption, OIDC on the control plane, WebAssembly plugins for custom middleware, and pluggable chat-history backends (PostgreSQL, Oracle, Redis, or in-memory)
• Observability stack: 90+ Prometheus metrics, OpenTelemetry tracing with W3C context propagation into engines, and structured JSON logs

Verdict

Worth evaluating if you run a heterogeneous fleet of self-hosted GPUs alongside cloud API keys and want one consistent API facade for the lot. Probably overkill if you are calling a single OpenAI endpoint from a monolith.