The vLLM project's answer to 'just run it in Kubernetes'

What it does

This is a Helm-based reference architecture that wraps vLLM engines in a Kubernetes-native serving layer. It adds a request router, Prometheus/Grafana observability, and hooks for KV-cache offloading so you can scale from a single pod to a distributed deployment without touching client code. The stack exposes the standard OpenAI API interface, making it a drop-in replacement for a standalone vLLM server.

The interesting bit

The router is the quiet star: it routes by session ID or prefix to maximize KV-cache reuse across backends, and the project is explicitly building toward KV-cache-aware algorithms and disaggregated prefill. That suggests the team is treating the inference fleet less like a stateless microservice and more like a coordinated cache hierarchy.

Key highlights

• Drop-in OpenAI API compatibility; clients need no changes to hit a distributed backend.
• Router supports round-robin, session-ID, and prefix-aware (WIP) routing with automatic Kubernetes service discovery.
• Bundled Grafana dashboard tracks TTFT, GPU KV-cache hit rates, pending/running requests, and instance health.
• Integrates with LMCache for KV-cache offloading.
• Roadmap includes autoscaling based on vLLM-specific metrics and a non-Python router rewrite for performance.

Caveats

• Prefix-aware routing is marked WIP, and several router improvements remain on the 2026 roadmap rather than in the current release.
• The project is tightly coupled to Kubernetes and Helm; there is no supported bare-metal or Docker Compose path visible in the sources.

Verdict

Worth a look if you are already running vLLM and need an official, opinionated way to move it to a multi-node Kubernetes fleet. Skip it if you are not on K8s or if you need a fully managed, autoscaling solution today—the autoscaler is still coming.