Your LLM's hidden layers are better bookies than its tokenizer

What it does

AnyJev wraps open LLMs so they answer structured questions—multiple choice, yes/no, scored ratings—with a probability distribution you can trust. It strips away position bias by averaging over cyclic option shifts at zero labels, then optionally fits a closed-form linear head on an intermediate hidden state using a few hundred examples. The result is a decision endpoint that costs less than one plain forward pass and emits odds instead of tokens.

The interesting bit

The project treats the LLM as a feature extractor, not a text generator: it often truncates the model to a middle block, finds that the remaining layers are just busy turning answers into tokens, and reads decisions from a pooling server serving those truncated hidden states. A fitted head is a few kilobytes, self-tunes on unlabelled traffic, and ships for specific model families—currently five Qwen3 variants.

Key highlights

• L0 needs no labels and cuts the rate of flipped answers when options are reversed from 0.230 to 0.073, while raising the share safely automatable at ≤5% error from 7.7% to 46.3%.
• L2 heads fit in seconds with 100–300 labels via closed-form solve; no backpropagation touches the base model.
• Deployed through vLLM as an embed server plus a ~100 KB head, with truncated checkpoints often running faster and slightly more accurately than full depth.
• Heads auto-adapt to reworded questions and shuffled options by re-estimating mean and scale from unlabelled traffic.
• Shipped heads cover five Qwen3 models; other architectures (Llama, Gemma, Mistral, DeepSeek) are on the roadmap.

Caveats

• Accuracy benchmarks on the included typed-decisions task measure agreement with a teacher LLM, not human ground truth.
• L2 heads are per-question and per-model: a head fit for one task won’t transfer to another, and only Qwen3 heads ship today.
• The letter-token readout caps choices at 26 options, and the project has not yet been evaluated inside live agent loops.

Verdict

Worth a look if you run LLM-based routing, scoring, or classification and need probabilities you can threshold for automation. Skip it if you need cross-task generalization or work outside the Qwen3 family and don’t want to generate your own labels.