Turing's last stand: 27B models on dual 2080 Ti, via a vLLM fork

What it does

It’s a fork of vLLM (tracking a recent upstream baseline) that keeps Turing SM75 GPUs — chiefly two 22 GB RTX 2080 Ti joined by NVLink — first-class citizens for local serving. The fork carries SM75-specific source patches, a tree of validated launch profiles, and an interactive launcher.sh that assembles hardware, model, weight format, and speculative-draft choices into a running service. Validated routes cover Qwen3.8 27B in FP8, NVFP4, and INT4, plus a 35B FP8 route, across TP=2 and TP=4 layouts.

The interesting bit

The pitch is arithmetic: two 22 GB 2080 Ti cards give you 44 GB of VRAM, 136 SMs, and 1,232 GB/s of bandwidth — by the README’s own table, more SMs, memory, and bandwidth than a single RTX 3090 Ti. The fork’s real product, though, is validation: the profiles encode known-good combinations of weight dtype, KV-cache dtype, context length, and draft model, so you’re not rediscovering them one CUDA-graph crash at a time.

Key highlights

• 220.84 tok/s single-request decode (4K prompt) on 2x 2080 Ti with Qwen 27B NVFP4, 256K context, FP8 KV; 209.35 tok/s at 32K.
• A 4x Tesla T10 FP8 route sustains ~190 tok/s with FP16 KV at 256K context.
• Supports TP, PP, and mixed TP/PP; validated on TP=2 (dual 2080 Ti, 2x T10) and TP=4 (4x T10).
• Four launch modes — normal, fast (the default), aggressive, safe — trade speed against risk.
• The wider SM75 family is in scope (TITAN RTX, Quadro RTX 6000/8000, Tesla T4/T10/T40), though those need independent validation.

Caveats

• The headline numbers come from synthetic inputs with high speculative-hit rates (DFlash2, K=7); the README itself warns that real-task throughput depends on draft acceptance and may not reach these figures.
• The 0.2.x line demands a very fresh toolchain — CUDA 13.0, PyTorch 2.13, GCC 15, Ubuntu 26.04+ — with older setups relegated to an unmaintained 0.1.x branch.
• NVLink (or at least verified PCIe P2P) is expected, 11 GB and 22 GB cards can’t be mixed in the 27B TP=2 routes, and some 4x T10 routes carry a noted image-semantic failure.

Verdict

If you own Turing cards — especially the 22 GB 2080 Ti pairs this is built around — this is the rare fork that has already done your homework. If you’re on Ampere or newer, upstream vLLM covers you, and the synthetic-input benchmarks won’t transfer.