← all repositories
OmniJev/Flide

One forward pass, many answers: Flide forks LLM state for decisions

A serving engine built for Jev-style decision models, which answers many questions about the same input by forking the KV state exactly instead of re-reading the prompt.

★550 stars Python Inference · Serving
Flide
Too new to rank. Velocity needs a few days of history.
not enough history yet
star history

What it does

Flide is inference infrastructure for “Jev-style” decision models — LLMs that take a state (an email, a support ticket, a scenario) and answer structured questions about it, like choosing among options or flagging urgency. It serves these via a /v1/systemone API and claims up to 9.5× speedups over vLLM and SGLang. The trick: instead of running the prompt through the model once per question, it forks the internal state so every question is answered in a single forward pass.

The interesting bit

The forking is exact — the README stresses that every answer is the one the model would give the question alone, i.e., no approximation from sharing state. There’s even a flide verify command that checks exact forking on your own model and requests, which is a refreshingly auditable claim for a speed-focused engine.

Key highlights

  • Supports a long roster of decision models (OneJev, GLiDE, Drex, Clef, pplx-decider, KnowLine, and many more), plus Qwen3.5 and Gemma 4 chat models.
  • Handles text and image inputs, LoRA adapters served as-is, and up to 256 options per question.
  • FP8 quantization (--quantize fp8) fits Gemma 4 12B on a single 24 GB card.
  • Email triage demo: 10,000 emails × 8 questions answered in 28 seconds on one H200 (10 ms per email).
  • Precompiled Triton kernels for RTX 40/50 series and RTX PRO 6000; Apache 2.0, Python 3.10+, PyTorch 2.12+.

Caveats

  • NVIDIA GPUs only, under Linux or Windows WSL2 — no CPU or AMD path mentioned.
  • The “Jev” ecosystem and its model families are niche; if you’re not already in that world, the model table won’t mean much to you.
  • Benchmark numbers are the project’s own (median latency on one H200), not independently reproduced in the README.

Verdict

If you’re serving decision-style models at scale — triage, classification, structured judgment — the exact-forking trick looks genuinely valuable and the speedups are large. If you just need a general chat/completions server, vLLM or SGLang remain the safer bet.

Frequently asked

What is OmniJev/Flide?
A serving engine built for Jev-style decision models, which answers many questions about the same input by forking the KV state exactly instead of re-reading the prompt.
Is Flide open source?
Yes — OmniJev/Flide is open source, released under the Apache-2.0 license.
What language is Flide written in?
OmniJev/Flide is primarily written in Python.
How popular is Flide?
OmniJev/Flide has 550 stars on GitHub.
Where can I find Flide?
OmniJev/Flide is on GitHub at https://github.com/OmniJev/Flide.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.