Buy accuracy with compute: a proxy that makes small models punch up

What it does OptiLLM is an OpenAI API-compatible proxy that wraps your existing LLM calls in 20+ inference-time optimization techniques — best-of-N, self-consistency, MCTS, Mixture of Agents, plan search, and friends. You pick a technique by prefixing the model name (e.g. moa-gpt-4o-mini) and the proxy does the rest: multiple calls, reflection, voting, or search behind one ordinary chat completion. No training, no fine-tuning, no changes to your client beyond the base URL.

The interesting bit It’s essentially a vending machine for test-time compute. The README’s benchmark table claims things like MARS lifting Gemini 2.5 Flash Lite from 43.3 to 73.3 on AIME 2025, and MOA on GPT-4o-mini matching GPT-4 on Arena-Hard-Auto — the trade is always the same: more API calls and tokens for more accuracy.

Key highlights

• 20+ approaches, from simple bon sampling to MCTS, Z3-backed logic solving, and the CePO pipeline from Cerebras
• Plugin layer adds extras: PII anonymization, URL fetching, code execution, structured JSON output, MCP client, and a learned router that picks the approach per prompt
• Multi-provider via LiteLLM — OpenAI, Anthropic, Google, Cerebras, 100+ models
• Docker variants include a fully offline image and a lightweight proxy-only one
• Runs as a plain drop-in; existing OpenAI clients work unchanged

Caveats

• The accuracy gains are bought with extra inference calls, so latency and token costs go up — the README doesn’t dwell on this, but it’s the whole business model
• Several techniques (Deep Confidence, CoT decoding, AutoThink) are marked N/A for proxy use, so the menu is smaller than the headline count suggests
• Benchmark numbers are self-reported per-technique; your task may not resemble AIME or LiveCodeBench

Verdict Worth a look if you have token budget to burn and want cheap models to reason like expensive ones. Skip it if you’re latency-sensitive or your tasks don’t benefit from sampling-and-voting tricks.