A tiny model trained to pick, not to chat — 22 ms per decision
Most 'pick one of these' LLM calls waste a generative model on a classifier's job — Jeff fine-tunes small open models to return a calibrated probability per option in a single forward pass, on your own hardware.

What it does
You describe a situation in plain words and list the options; Jeff replies with a calibrated probability for each, plus its pick and a confidence — no generated text, no parsing. Three question types cover the ground: choice (up to 254 options in the v1.1 Qwen models), noul (yes/no as a probability), and score (a point on a scale you describe), with several independent questions answered in one request. Under the hood it’s a family of Qwen3.5 and Gemma 4 fine-tunes (0.8B–2B, weights on Hugging Face) serving a local HTTP endpoint in Jev’s request format — same API, explicitly no affiliation with Jev’s makers. A decision costs about 22 ms on an RTX PRO 6000 and 28 ms on an Apple M4 Max, with PyTorch and MLX backends.
The interesting bit
The self-awareness is the charm: the endpoint is literally /v1/systemone, and the README is blunt that this is a snap-judgment model that trails Jev on reasoning-heavy benchmarks while matching or beating it on classification. The provenance is just as unusual — all training data is synthetic, written by an open model, with a closed model used only to spot-check a sample, and training ran on a single workstation GPU.
Key highlights
- Zero-shot by construction: your categories never need to appear in the training data; you describe them in words and Jeff picks.
- On 4,599 questions across five public benchmarks, the 0.8B scores 79.1% and the 2B 82.0% overall, against Jev’s published 83.0% — ahead on Financial PhraseBank (95.7 vs 77.0) and RAGTruth (85.6 vs 77.3).
- The games are a fun stress test: zero-shot, the 0.8B matches a hand-coded rule bot at Doom (6.55 kills each), edges it at Frogger, and trails at Pac-Man, deciding in 29–49 ms per move on an M4 Max.
- Fine-tuning pays fast: a chess fine-tune (600,000 Stockfish-labelled Lichess positions, ~3½ hours on one GPU) lifted held-out puzzles from 15.5% to 55.8% and kept up with ~600 simultaneous blitz games; a voice-navigation set went from 31.7% to 95.8% in under half an hour.
- v1.1 stretched
choicefrom 26 to 254 options and lifted the 0.8B’s long-list accuracy from 40% to 95%.
Caveats
- Reasoning is explicitly not the job: BBH lands at 64.9 vs Jev’s 94.3, and JevBench hard at 46.7 vs 73.3.
- The 254-option upgrade cost the 2B a hair (83.1% → 82.0% overall), and the Gemma variant is still v1.0, capped at 26 options.
- The Jev and AutoJev comparison figures were measured on a different sample of the same benchmarks — a close race, not a controlled one.
Verdict If you route tickets, label moderation queues, or triage anything where latency and calibration matter more than deep reasoning, this is a useful shape: small, local, fine-tunable in an afternoon on one GPU. If the task needs actual reasoning, keep the bigger model — Jeff doesn’t pretend otherwise, and neither does its README.
Frequently asked
- What is firelex/jeff?
- Most 'pick one of these' LLM calls waste a generative model on a classifier's job — Jeff fine-tunes small open models to return a calibrated probability per option in a single forward pass, on your own hardware.
- Is jeff open source?
- Yes — firelex/jeff is open source, released under the MIT license.
- What language is jeff written in?
- firelex/jeff is primarily written in Python.
- How popular is jeff?
- firelex/jeff has 1.6k stars on GitHub.
- Where can I find jeff?
- firelex/jeff is on GitHub at https://github.com/firelex/jeff.