Your LLM classifier is mostly a very expensive lookup table

What it does TRACER trains lightweight classical models to mimic your LLM on easy classification inputs. A calibrated gate routes predictable traffic locally and defers edge cases back to the LLM, while a parity guard blocks deployment if the surrogate can’t match your threshold on held-out data.

The interesting bit It treats your LLM as a slow, expensive labeling oracle. Deferred calls feed new traces into the next refit, so coverage grows without human tuning.

Key highlights

• Claims 92.2% local coverage and 96.1% teacher agreement on Banking77, with projected annual savings of $302,850 at 10k queries/day.
• Surrogate inference is CPU-bound and sub-millisecond.
• Includes local OpenTelemetry observability and a free optional cloud dashboard.
• JavaScript routing requires a Python sidecar.

Caveats

• The Banking77 result used heavier, opt-in tree models; the default linear + MLP may lag on complex many-class tasks.
• The pre-fit scan wants ~1,000 traces for a reliable estimate.

Verdict Grab it for high-volume LLM classification; skip for low-volume or generative reasoning tasks.