An RTX 3090 should not run a 27B model this fast

What it does This project ships a containerized, pre-patched vLLM setup built around one very specific goal: serving the 27-billion-parameter Qwen3.8 model on a single RTX 3090. It offers two personalities — a batch mode that chases aggregate throughput (~1,035 tok/s at 64 concurrent requests via int8 tensor-core GEMMs) and a single-user mode that chases per-request latency with speculative decoding tricks. Everything is wrapped in a Docker image with requantization scripts and an OpenAI-compatible API.

The interesting bit The speed does not come from better hardware; it comes from treating the GPU’s 24 GB budget like a puzzle box. The author replaces standard vLLM internals with int8 GEMMs, a calibrated int4 lm_head, and custom draft vocabularies pulled from the model’s own outputs. For single-user chat there is MTP speculation; for document-quoting workloads there is DFlash2, which drafts up to seven tokens in one pass and can fill verify blocks straight from the prompt context. A KVarN patch even stretches the KV cache to roughly 268k tokens, though that particular trick trades precision for space.

Key highlights

• Two tuned profiles: batch mode for API backends (~1,035 tok/s aggregate decode) and single-user mode for chat (~121 tok/s with MTP, up to ~382 tok/s when reproducing long documents with DFlash2).
• Prefix caching that preserves both attention KV and recurrent DeltaNet state, cutting second-turn warm-up over a 25k document from 22 seconds to half a second.
• Lossless speculative decoding (MTP and DFlash2) that samples the same distribution as non-speculative generation, at least when not using the KVarN context mode.
• Pre-built container image with all patches and verification gates baked in; the model is downloaded and requantized automatically on first boot.
• Optional KVarN integration pushes context capacity to roughly 268k tokens by using a lossy 4/2-bit KV cache.

Caveats

• SPEC=dflash2 is strictly a single-user optimization; it burns scarce recurrent-state pages, so concurrency collapses if you try to serve multiple chat sessions with it.
• Raising DFLASH_TOKENS to 15 nearly doubles the per-request recurrent-state footprint, which halves available request slots and trims context length — the README notes this is only worth enabling for verbatim document reproduction, not chat.
• The KVarN “huge context” mode is explicitly lossy, unlike the rest of the stack, and WSL2 users face a roughly 20% performance penalty plus extra pinned-memory configuration.

Verdict Grab this if you own an RTX 3090 (or equivalent 24 GB card) and want to turn it into a local Qwen3.8-27B server without renting an A100. Skip it if you need multi-user speculative decoding, are allergic to Docker and custom vLLM patches, or expect Windows to match bare-metal Linux speeds.