Long-context LLM serving that treats heads differently

What it does RedKnot is an SGLang extension that speeds up long-context LLM inference by classifying every (layer, kv_head) pair into one of four buckets—global, local, retrieval, or dense—and applying a tailored KV storage and reuse policy to each. It also skips feed-forward computation for low-importance tokens and maintains per-head page tables so different classes can see different context windows. The project reports roughly 50–70% prefill FLOP savings and 1.35×–3.2× TTFT speedups on benchmarks, with accuracy that is near-lossless or occasionally better than the dense baseline.

The interesting bit Rather than treating the KV cache as a monolithic block, RedKnot stores reusable segments offline and selectively recomputes only necessary tokens at serving time, using RoPE relocation to keep the math aligned. It is essentially a bet that long-context attention is sparser than it looks, and that some tokens deserve the full FFN while others do not.

Key highlights

• Four-class head taxonomy: global, local, retrieval, and dense, each with its own KV visibility and reuse rules.
• Offline KV reuse with RoPE relocation: segments are cached offline and recalled with numerical alignment guarantees.
• Elastic sparsity: a token-selective FFN that skips computation for tokens deemed low-importance.
• SegPagedAttention: per-head page tables and segmented KV stores allowing different visible windows per head class.
• Reported gains: up to ~3.2× TTFT speedup on Mistral-7B and ~70% FLOP reduction on Qwen3-32B, with F1 scores matching or exceeding dense baselines in the provided tests.

Caveats

• Llama-3.3-70B-Instruct suffers from decode degradation (repeated tokens) and memory/cross-device errors under long-context evaluation; the authors note this is a pending algorithm and config issue.
• Qwen3.5-MoE accuracy drops at longer contexts in reported LongBench tests (for example, F1 falls from 0.792 to 0.576 at 32K).
• DeepSeek-V4 and the full Qwen 3.5 series are not yet fully adapted in the current open-source release.

Verdict A solid experiment if you are already on SGLang and serving long-context RAG or document workloads. Avoid if you need a model-agnostic, production-ready drop-in today.