A Swedish mill for grinding KV caches smaller—and faster

What it does KVarN replaces vLLM’s standard KV-cache storage with a custom attention backend that compresses keys to 4-bit and values to 2-bit using Triton kernels compiled at runtime. The goal is to stretch context capacity by several multiples without the usual throughput collapse or accuracy drift, and it requires no calibration step or model changes.

The interesting bit The method allocates its bit budget unevenly across keys and values, then runs each tile through a Hadamard rotation and an iterative variance-normalization pass—a Sinkhorn-style massage that spreads outliers before rounding. According to the project’s benchmarks, this lands Qwen3-32B in an upper-right corner TurboQuant cannot reach: FP16-level accuracy, higher throughput, and roughly 4× the KV capacity.

Key highlights

• Native vLLM fork (v0.23.0) with JIT-compiled Triton kernels, not a wrapper script.
• Supports dense transformers, MLA models such as GLM-4.7-Flash, and hybrid Mamba/linear-attention architectures.
• Compatible with speculative decoding (MTP, DFlash) and weight quantization like AWQ INT4.
• Two tile sizes: g128 for maximum capacity and g64 for finer granularity.
• On MLA models it quantizes the compressed latent directly; on hybrid models it leaves recurrent states untouched.

Caveats

• You adopt the project’s entire forked engine rather than installing a plugin into an existing vLLM setup.
• On memory-tight single-GPU deployments, vLLM’s CUDA-graph profiler can over-reserve memory and shrink the KV pool; the README notes you may need to disable it to recover the full advertised capacity.
• The throughput-above-FP16 claim is demonstrated on Qwen3-32B with tensor parallelism; MLA models like GLM-4.7-Flash show throughput slightly below (0.94×) bf16, so the speed win is architecture-dependent.

Verdict Worth testing if you operate long-context or agentic vLLM workloads and are memory-bound on the KV cache. Skip it if you are already comfortable with FP16 capacity or if maintaining a forked inference stack is a dealbreaker.