Speculative inference built for the 'I only have one GPU' crowd

What it does

Lucebox Hub is a C++ inference server that runs LLMs locally via an OpenAI-compatible HTTP API. It ships custom CUDA and HIP kernels—organized into self-contained optimizations like Megakernel, DFlash, and PFlash—each tuned for specific model families (mostly Qwen and Gemma) and hardware targets from the RTX 2080 Ti up to the RTX 5090. The server uses GGUF weights and supports speculative decode, speculative prefill compression, quantized KV caches, and multi-GPU draft offloading.

The interesting bit

Instead of one generic engine, every optimization is a bespoke suit: the PFlash prefill drafter uses a tiny 0.6B model to compress prompts, while DDTree speculative decoding verifies draft tokens in a tree structure. The README claims a ~5.6× combined speedup on Qwen 3.6-27B with PFlash against a vendored llama.cpp baseline, though those numbers are tied to exact quantization and KV cache settings.

Key highlights

• Speedups measured against llama.cpp with matching KV quant: up to ~5.6× for Qwen 3.6-27B + PFlash, 4.84× with DDTree, and ~2.6× on AMD HIP (Strix Halo / RX 7900 XTX).
• Runs on NVIDIA Ampere through Blackwell (and back to Turing), plus an AMD HIP backend; RTX 3090 is the reference bench target.
• OpenAI-compatible HTTP server with client harnesses for Claude Code, Codex, Open WebUI, and others.
• Aggressive memory management: TQ3_0 KV cache quant (3.5 bits per value), sliding attention windows, draft residency controls, and disk-persisted prefix caches.
• No PyTorch required for the main DFlash server; only the Megakernel component needs it.

Caveats

• Most headline speedups are specific to Qwen and Gemma families; the README explicitly states each optimization targets a specific model family and hardware generation.
• Several GPU tiers (RTX 4090 via WSL2, Jetson AGX Thor, Volta/Pascal) are marked as community-tested or unbenched, so your mileage may vary.
• Prefill compression and tree-verify modes require separate draft GGUF weights published on the project’s Hugging Face hub.

Verdict

Worth a look if you’re trying to host 27B-class models on a single consumer GPU and need an OpenAI-shaped drop-in replacement. Skip it if you need broad model-agnostic optimization or production guarantees on untested hardware.