Ops glue for 1M-token DeepSeek on two DGX Sparks

What it does

This repository is an infrastructure recipe for serving deepseek-ai/DeepSeek-V4-Flash-0731 across two DGX Spark nodes using vLLM with tensor parallelism and DSpark speculative decoding. It wraps container orchestration, NCCL networking setup, and dozens of environment-variable hotfixes into shell scripts so you don’t have to hand-tune nvfp4_ds_mla KV caches or RoCE interfaces yourself. The default profile targets a 1M-token context ceiling with six concurrent sequences, though it also documents a separate Stage-C path for a 200K-context, 16-slot high-throughput lane.

The interesting bit

The repo’s real value is its bug-fix inventory: patches for issues #22, #26, #27, #31, and others are applied via container-startup hooks and env vars rather than upstream merges, effectively turning the deployment into a curated compatibility layer between a specific checkpoint and a very specific two-node NVFP4 cluster.

Key highlights

• Serves DeepSeek V4 Flash 0731 at up to 1M tokens (MAX_MODEL_LEN=1048576) with speculative decoding (MTP_NUM_TOKENS=5).
• Targets 2x DGX Sparks via RoCE/NCCL with TP=2 and nvfp4_ds_mla KV caching.
• Bundles hotfixes for long-context decode, prefill chunking, P-core spin-wait, and thinking-token budgets as env-var toggles.
• Optional Qwen3-VL sidecar on port 8889 with MCP harness.
• Benchmarked aggregate throughput around 160–190 tok/s for six short-chat streams on the default Anemll image.

Caveats

• This is not generic infrastructure: you need two DGX Sparks, matching HF caches on both nodes, and working RoCE/NCCL fabric.
• Six simultaneous 1M-token requests won’t fit; the KV pool limits concurrency, and six cold 32K–128K prefills will queue and starve decode down to roughly 8 tok/s.
• Several flags (e.g., VLLM_DSPARK_GPU_REJECTED_CONTEXT_MASK) are no-ops on the default Anemll image and only matter if you switch to the Stage-C container path.

Verdict

Grab this if you actually own two DGX Sparks and want to run DeepSeek V4 Flash without debugging NCCL topology yourself. Everyone else should treat it as a detailed reference for vLLM-at-scale pain points rather than something to clone.