One GPU, one model, zero fallbacks: a 4× faster LLM server

What it does halogen-flash-server is an OpenAI-compatible inference host for Qwen3.8-Flash-Next that runs exclusively on AMD Strix Halo (gfx1151). It discards general-purpose runtimes in favor of kernels hand-written for that specific GPU and model family, with the stated goal of shrinking wall-clock time on long prompts. Since 0.7.0 it can also ingest off-the-shelf llama.cpp GGUF files and push them through the same specialized kernels.

The interesting bit The engine treats portability as a liability: there are no abstraction layers, no fallback paths, and no support for other silicon. Speculative decoding is verified to be byte-identical to greedy decode at temperature 0 on every release, using both the model’s own draft head and the request’s prompt text as draft sources—so it functions as a pure speed optimization rather than a quality trade.

Key highlights

• Claims roughly 4× faster end-to-end latency than published alternatives for this model on Strix Halo, while running at 5.53 bits per weight—higher precision than most competing runs.
• Exposes an OpenAI-shaped API (/v1) with optional vision, explicit 400 errors for unsupported sampling fields, and server-side defaults that can be set without restarting.
• Opens existing llama.cpp GGUFs directly—such as unsloth’s UD-IQ4_XS—with no conversion step.
• Supports up to 1M token context, attention budgets, and a preview of composable context.
• Can operate fully offline after weights are cached, and optionally splits the API frontend from the inference backend.

Caveats

• The decode speed comparison is deliberately softer than the prefill claim; the README notes that two of three competitors do not clearly state whether speculative decoding was enabled at 32K context.
• The API and engine containers must use the exact same image tag, or they may miscommunicate features such as image placeholders.
• It is welded to one GPU architecture and one model family; if you do not own Strix Halo, there is no compatibility path.

Verdict If you have Strix Halo hardware and want the lowest possible local latency for this specific 179B-parameter MoE, the project is built precisely for you. Anyone without that exact AMD chip should ignore it—the authors are explicit that portability was the price of admission.