A toolbox of scorers for judging LLM output without hand-rolling prompts

What it does

Autoevals gives you ready-made evaluators for grading AI outputs: model-graded judges for subjective tasks like factuality, humor, moderation, and summarization, plus simpler scorers like Levenshtein distance, exact match, BLEU-style statistics, and embedding similarity. Many of the LLM-judge prompts are adapted from OpenAI’s evals project, but reworked so you can run them on single examples, tweak the prompts, and inspect the rationale behind each score. It ships in both Python and TypeScript.

The interesting bit

The library treats evaluation prompts as templates (e.g. templates/factuality.yaml) rather than opaque black boxes — you can see, edit, and debug exactly what the judge is being asked. There’s also a LLMClassifier primitive for building your own judges from a prompt template plus a choice-to-score mapping, which is a surprisingly clean way to express “pick the better of two options” style evals.

Key highlights

• Broad scorer catalog: factuality, security, SQL, translation, moderation, plus a full RAG suite (context recall, faithfulness, answer relevancy, etc.)
• Works with any OpenAI-compatible API via OPENAI_BASE_URL, or the Braintrust Gateway for multi-provider routing and caching
• Returns a 0–1 score along with metadata like the judge’s rationale, not just a verdict
• Custom judges are first-class: bring your own prompt, scoring map, and chain-of-thought toggle
• Sync and async APIs in both languages

Caveats

• The default path routes through the Braintrust Gateway (free while in beta) — using other providers or a plain OpenAI client requires explicit configuration
• The Braintrust logging integration is optional, but the README’s structure makes the platform tie-in fairly prominent

Verdict

If you’re building eval harnesses and want battle-tested judge prompts without writing them from scratch, this is a solid grab-bag. If you need fully vendor-neutral tooling with zero gateway involvement, expect a bit of configuration to steer it elsewhere.