One forward pass, many answers: Flide forks LLM state for decisions
A serving engine built for Jev-style decision models, which answers many questions about the same input by forking the KV state exactly instead of re-reading the prompt.
What it does
Flide is inference infrastructure for “Jev-style” decision models — LLMs that take a state (an email, a support ticket, a scenario) and answer structured questions about it, like choosing among options or flagging urgency. It serves these via a /v1/systemone API and claims up to 9.5× speedups over vLLM and SGLang. The trick: instead of running the prompt through the model once per question, it forks the internal state so every question is answered in a single forward pass.
The interesting bit
The forking is exact — the README stresses that every answer is the one the model would give the question alone, i.e., no approximation from sharing state. There’s even a flide verify command that checks exact forking on your own model and requests, which is a refreshingly auditable claim for a speed-focused engine.
Key highlights
- Supports a long roster of decision models (OneJev, GLiDE, Drex, Clef, pplx-decider, KnowLine, and many more), plus Qwen3.5 and Gemma 4 chat models.
- Handles text and image inputs, LoRA adapters served as-is, and up to 256 options per question.
- FP8 quantization (
--quantize fp8) fits Gemma 4 12B on a single 24 GB card. - Email triage demo: 10,000 emails × 8 questions answered in 28 seconds on one H200 (10 ms per email).
- Precompiled Triton kernels for RTX 40/50 series and RTX PRO 6000; Apache 2.0, Python 3.10+, PyTorch 2.12+.
Caveats
- NVIDIA GPUs only, under Linux or Windows WSL2 — no CPU or AMD path mentioned.
- The “Jev” ecosystem and its model families are niche; if you’re not already in that world, the model table won’t mean much to you.
- Benchmark numbers are the project’s own (median latency on one H200), not independently reproduced in the README.
Verdict
If you’re serving decision-style models at scale — triage, classification, structured judgment — the exact-forking trick looks genuinely valuable and the speedups are large. If you just need a general chat/completions server, vLLM or SGLang remain the safer bet.
Frequently asked
- What is OmniJev/Flide?
- A serving engine built for Jev-style decision models, which answers many questions about the same input by forking the KV state exactly instead of re-reading the prompt.
- Is Flide open source?
- Yes — OmniJev/Flide is open source, released under the Apache-2.0 license.
- What language is Flide written in?
- OmniJev/Flide is primarily written in Python.
- How popular is Flide?
- OmniJev/Flide has 550 stars on GitHub.
- Where can I find Flide?
- OmniJev/Flide is on GitHub at https://github.com/OmniJev/Flide.