One Rust gateway to rule all your LLM providers

What it does

TensorZero is an open-source LLMOps platform that fronts nearly every major LLM provider through a single Rust gateway, then adds observability, evaluation, prompt optimization, and experimentation on top. It stores inferences and feedback in your own database, benchmarks workflows with heuristics or LLM judges, and runs adaptive A/B tests to find the best prompts and models. The gateway claims sub-millisecond p99 latency overhead at over 10,000 QPS.

The interesting bit

Rather than forcing a rip-and-replace, TensorZero is designed for incremental adoption—you can start with just the gateway, then layer in the feedback loop that turns production traffic into fine-tuning datasets and automated prompt-engineering runs. The README also notes, somewhat boldly, that the project currently fuels roughly 1% of global LLM API spend.

Key highlights

• Unified Rust gateway supporting Anthropic, OpenAI, Azure, Bedrock, DeepSeek, vLLM, Groq, and any OpenAI-compatible endpoint
• Observability UI and database storage for inferences and feedback, with OpenTelemetry and Prometheus exports
• Evaluation suite treating individual inferences like unit tests and end-to-end workflows like integration tests
• Optimization via supervised fine-tuning, RLHF, GEPA prompt engineering, and dynamic in-context learning
• “Autopilot” mode that automates eval setup, prompt optimization, and A/B testing across LLM agents

Caveats

• Several features are marked “Soon” in the README, including AI-assisted debugging, synthetic data generation, and headless evaluations
• The ~1% of global LLM API spend figure is stated without visible methodology in the provided sources
• Autopilot’s performance claims are illustrated with a bar chart but lack detailed reproducibility notes in the README

Verdict

Production teams tired of gluing together separate gateways, eval frameworks, and optimizers will find the integrated flywheel useful; developers making sporadic API calls should probably stick to their current setup.