Measuring LLM hallucinations without ground truth

What it does

UQLM is a Python library that turns various uncertainty-quantification techniques into a single confidence score between 0 and 1 for each LLM response. A higher score means the model is less likely to be hallucinating. It plugs into LangChain chat models and bundles five families of scorers: black-box consistency checks, white-box token-probability metrics, LLM-as-a-Judge panels, ensembles, and claim-level long-text scorers.

The interesting bit

Instead of fact-checking against a database, UQLM treats hallucination detection as a measurement of statistical self-doubt. You can flag dubious answers even when you do not know the correct answer yourself—by watching how much the model contradicts its own resampled outputs or by reading its token-level probabilities.

Key highlights

• Black-box scorers are universal—they work with any LLM API—but they add medium-to-high latency and cost because they sample multiple responses.
• White-box single-generation scorers add almost no overhead, since they reuse token probabilities already returned by the model; the catch is that many APIs lock those probabilities away.
• LLMPanel lets you assemble a panel of judge LLMs to vote on the reliability of the primary model’s answer.
• A mitigation mode (use_best=True) automatically picks the response with the lowest uncertainty instead of just scoring it.
• The package implements published methods including Semantic Entropy, BERTScore, P(True), and CoCoA.

Caveats

• White-box scorers require API access to token probabilities, which not every provider exposes.
• Black-box and long-text scorers demand multiple LLM calls, so the cost and latency scale with each evaluation.
• The README does not provide benchmark numbers showing how closely each scorer’s confidence aligns with real-world hallucination rates.

Verdict

Useful if you are shipping LLM applications and need an automated filter for low-confidence outputs before they reach users. Less useful if you have a ground-truth oracle handy or if your inference budget is too tight to tolerate extra sampling.