Rescuing the V100 from software neglect

What it does

1Cat-vLLM is a vLLM fork that optimizes modern large language model inference specifically for NVIDIA Tesla V100 and SM70 (Volta) hardware. It rebuilds attention kernels, precision contracts, and speculative decoding so that 2017-era GPUs can run current models like Qwen3.8 and DeepSeek-V4-Flash with usable throughput. The project reports end-to-end benchmarks tied to actual coding and reasoning scores, not just isolated kernel metrics.

The interesting bit

Most inference stacks have moved on from Volta to Ampere and Hopper, but this fork extracts roughly 3.4× more useful attention TFLOP/s from V100s by packing GQA heads into wider Tensor-Core GEMMs and repairing FP32 attention states for FP8 KV caches. It also refuses to play the benchmark game: every throughput claim is paired with dataset pass rates, natural-stop health, and tool-calling accuracy so the speed figures actually mean something.

Key highlights

• Long-context attention compute improved from ~17.9 to ~60.8 useful TFLOP/s on the same V100 hardware, with an experimental research ceiling near 79 TFLOP/s.
• A 4× V100 16GB configuration pushes roughly 260 tok/s on Qwen3.8-27B-NVFP4 with DFlash2 in demo conditions, and sustains over 200 tok/s on practical coding workloads while maintaining task accuracy.
• Benchmarks bundle raw throughput with quality gates: dataset scores, natural-stop rates, output-cap failures, and speculative-decoding acceptance are all reported together.
• Supports modern quantization formats including NVFP4 and mixed FP8, plus speculative decoding variants like MTP4 and adaptive lookup, all on hardware the upstream stack largely ignores.

Caveats

• The headline ~260 tok/s figure is a specific real-machine demo, not a universal fixed decode rate; throughput varies significantly with model, context length, batch size, and KV cache dtype.
• Some peak figures, such as the ~79 TFLOP/s attention ceiling and the 316 tok/s repeated-context lookup mode, are experimental or opt-in contracts rather than default release quality.
• The README explicitly notes that some coding benchmark failures are caused by long reasoning exhausting a 16K output budget, not by model errors, which complicates simple speed-vs-quality comparisons.

Verdict

Grab this if you are sitting on a pile of V100s and want to run current models without treating the hardware as landfill. If you are already on H100s or newer, it is a fascinating optimization study but not your production path.