Training speculative decoders that actually talk to SGLang

What it does SpecForge trains speculative decoding draft models and exports them to the SGLang serving engine. It supports distributed training flavors—FSDP, tensor parallelism, online and offline modes—and targets the common frustration that most speculative decoding code is either abandoned or incompatible with production serving stacks.

The interesting bit Instead of leaving you with a research checkpoint and a prayer, SpecForge ships a maintained pipeline into SGLang. The team also releases production-grade draft models as SpecBundle, and the framework itself trained the Eagle3 draft model used for GPT-OSS.

Key highlights

• Distributed training via FSDP, tensor parallelism, and online/offline strategies.
• Native SGLang compatibility; the README emphasizes zero extra porting effort.
• Bundled SpecBundle checkpoints claim up to 4× inference speedup when paired with SGLang.
• Actively maintained under LMSYS with infrastructure from Voltage Park.
• Builds explicitly on prior art from EAGLE, BaldEagle, and NVIDIA’s TensorRT-Model-Optimizer.

Caveats

• The README is mostly release notes and links; concrete architecture details live in external documentation.
• The “effortless port” is strictly to SGLang, so teams on vLLM or TGI gain less from the serving integration.

Verdict A sensible choice if you are already in the SGLang orbit and need to train or fine-tune draft models. Everyone else should treat it as a well-curated training reference rather than a drop-in serving upgrade.