Is heterogeneous-gpu-pd-lab open source?

Yes — Soulmate-Halo/heterogeneous-gpu-pd-lab is an open-source project tracked on heatdrop.

What language is heterogeneous-gpu-pd-lab written in?

Soulmate-Halo/heterogeneous-gpu-pd-lab is primarily written in C++.

How popular is heterogeneous-gpu-pd-lab?

Soulmate-Halo/heterogeneous-gpu-pd-lab has 1.1k stars on GitHub.

Where can I find heterogeneous-gpu-pd-lab?

Soulmate-Halo/heterogeneous-gpu-pd-lab is on GitHub at https://github.com/Soulmate-Halo/heterogeneous-gpu-pd-lab.

← all repositories

Soulmate-Halo/heterogeneous-gpu-pd-lab

Your RTX 3080 can pull prefill duty for a DGX Spark

A lab of real-hardware experiments pairing cheap dense compute cards with big-memory hosts to split prefill and decode across mismatched GPUs.

★1.1k stars C++ Inference · Serving

View on GitHub ↗

Collecting fresh signals — velocity needs a few days of history.

collecting data…

star history

What it does

This repo documents heterogeneous GPU deployments where a fast, small-VRAM “dense” card (RTX 3080, RTX 6000D) accelerates a large-memory but slower host (DGX Spark, AI Max+ 395, M3 512G). The core trick is prefill/decode separation plus MoE expert placement: hot experts and the dense main path run on the fast card, cold experts run on the big-memory side, with KV and expert traffic crossing hosts over RDMA/NIXL. It’s a bundle of deployment configs, patches, Docker images, and benchmark evidence rather than a single library.

The interesting bit

The benchmarks are refreshingly honest: all numbers are untuned raw measurements, and the README explicitly warns against comparing them to tuned figures from other projects. It also openly discloses reproduction limits — e.g., the frozen HTTP router can’t hand a single request from Spark prefill to 3080 decode, so joint decode is RDMA expert cooperation, not a true PD handoff.

Key highlights

Qwen3.8 Flash on RTX 3080 + AI Max 395: prefill jumps ~3.1x (273 → 844 tok/s); decode stays flat since it runs on the 395 either way.
Qwen3.8 Flash V2 on 3080 + DGX Spark: reported ~2.25x aggregate and ~2.78x single-stream decode gains over Spark solo.
DS4.1 Flash V8: two RTX 6000D cards in vLLM TP2 fronting four Spark nodes in SGLang TP4/EP4, sustaining 16,000+ tok/s batch prefill at C16–C32.
DS4.1 Flash V10: a single 6000D handles all prefill for four Spark decode nodes, with KV crossing hosts over NIXL/RoCE.
Docker images ship configs, patches, and verification scripts — but deliberately exclude the multi-hundred-GB weights and private role images.

Caveats

Several benchmark comparisons are apples-to-oranges: different input lengths, concurrency, and workloads across setups; the README itself says treat them as order-of-magnitude references only.
The V2 decode results came without raw logs, concurrency, or workload details in that revision.
Reproducing anything here requires serious hardware (multiple hosts, RDMA, specific GPUs) plus separately-sourced weights — this is a lab, not a turnkey product.

Verdict

Worth a close read if you’re building multi-GPU or MoE inference on mixed hardware and want real, unpolished numbers. If you were hoping for a drop-in inference server, this isn’t it — it’s an evidence trail for people who like wiring GPUs together.

Frequently asked

What is Soulmate-Halo/heterogeneous-gpu-pd-lab?: A lab of real-hardware experiments pairing cheap dense compute cards with big-memory hosts to split prefill and decode across mismatched GPUs.
Is heterogeneous-gpu-pd-lab open source?: Yes — Soulmate-Halo/heterogeneous-gpu-pd-lab is an open-source project tracked on heatdrop.
What language is heterogeneous-gpu-pd-lab written in?: Soulmate-Halo/heterogeneous-gpu-pd-lab is primarily written in C++.
How popular is heterogeneous-gpu-pd-lab?: Soulmate-Halo/heterogeneous-gpu-pd-lab has 1.1k stars on GitHub.
Where can I find heterogeneous-gpu-pd-lab?: Soulmate-Halo/heterogeneous-gpu-pd-lab is on GitHub at https://github.com/Soulmate-Halo/heterogeneous-gpu-pd-lab.