← all repositories
firelex/jeff

A tiny model trained to pick, not to chat — 22 ms per decision

Most 'pick one of these' LLM calls waste a generative model on a classifier's job — Jeff fine-tunes small open models to return a calibrated probability per option in a single forward pass, on your own hardware.

★1.6k stars Python Language ModelsML Frameworks
jeff
Not currently ranked.
star history

What it does You describe a situation in plain words and list the options; Jeff replies with a calibrated probability for each, plus its pick and a confidence — no generated text, no parsing. Three question types cover the ground: choice (up to 254 options in the v1.1 Qwen models), noul (yes/no as a probability), and score (a point on a scale you describe), with several independent questions answered in one request. Under the hood it’s a family of Qwen3.5 and Gemma 4 fine-tunes (0.8B–2B, weights on Hugging Face) serving a local HTTP endpoint in Jev’s request format — same API, explicitly no affiliation with Jev’s makers. A decision costs about 22 ms on an RTX PRO 6000 and 28 ms on an Apple M4 Max, with PyTorch and MLX backends.

The interesting bit The self-awareness is the charm: the endpoint is literally /v1/systemone, and the README is blunt that this is a snap-judgment model that trails Jev on reasoning-heavy benchmarks while matching or beating it on classification. The provenance is just as unusual — all training data is synthetic, written by an open model, with a closed model used only to spot-check a sample, and training ran on a single workstation GPU.

Key highlights

  • Zero-shot by construction: your categories never need to appear in the training data; you describe them in words and Jeff picks.
  • On 4,599 questions across five public benchmarks, the 0.8B scores 79.1% and the 2B 82.0% overall, against Jev’s published 83.0% — ahead on Financial PhraseBank (95.7 vs 77.0) and RAGTruth (85.6 vs 77.3).
  • The games are a fun stress test: zero-shot, the 0.8B matches a hand-coded rule bot at Doom (6.55 kills each), edges it at Frogger, and trails at Pac-Man, deciding in 29–49 ms per move on an M4 Max.
  • Fine-tuning pays fast: a chess fine-tune (600,000 Stockfish-labelled Lichess positions, ~3½ hours on one GPU) lifted held-out puzzles from 15.5% to 55.8% and kept up with ~600 simultaneous blitz games; a voice-navigation set went from 31.7% to 95.8% in under half an hour.
  • v1.1 stretched choice from 26 to 254 options and lifted the 0.8B’s long-list accuracy from 40% to 95%.

Caveats

  • Reasoning is explicitly not the job: BBH lands at 64.9 vs Jev’s 94.3, and JevBench hard at 46.7 vs 73.3.
  • The 254-option upgrade cost the 2B a hair (83.1% → 82.0% overall), and the Gemma variant is still v1.0, capped at 26 options.
  • The Jev and AutoJev comparison figures were measured on a different sample of the same benchmarks — a close race, not a controlled one.

Verdict If you route tickets, label moderation queues, or triage anything where latency and calibration matter more than deep reasoning, this is a useful shape: small, local, fine-tunable in an afternoon on one GPU. If the task needs actual reasoning, keep the bigger model — Jeff doesn’t pretend otherwise, and neither does its README.

Frequently asked

What is firelex/jeff?
Most 'pick one of these' LLM calls waste a generative model on a classifier's job — Jeff fine-tunes small open models to return a calibrated probability per option in a single forward pass, on your own hardware.
Is jeff open source?
Yes — firelex/jeff is open source, released under the MIT license.
What language is jeff written in?
firelex/jeff is primarily written in Python.
How popular is jeff?
firelex/jeff has 1.6k stars on GitHub.
Where can I find jeff?
firelex/jeff is on GitHub at https://github.com/firelex/jeff.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.