A pipeline schedule that overlaps forward, backward, and networking

What it does

DualPipe is a scheduler for pipeline-parallel GPU training that sends micro-batches through ranks in both directions at the same time. It interleaves forward chunks, backward chunks, and weight-gradient communication so they share time windows rather than queueing sequentially. The repo also includes DualPipeV, a folded variant that preserves the same bubble reduction with fewer pipeline-parallel devices.

The interesting bit

The README lays out the trade-offs in a comparison table: versus classic 1F1B, DualPipe roughly halves the bubble penalty, but stores 2× parameters per device and one extra activation buffer. DualPipeV keeps the identical bubble formula while halving device count—a reminder that the most expensive part of training is often the real estate you leave idle.

Key highlights

• Bidirectional scheduling overlaps forward and backward execution on the same stages
• Ships with both the full DualPipe schedule and the condensed DualPipeV variant
• Requires PyTorch 2.0+ and a user-written overlapped_forward_backward method for real models
• Extracted from the DeepSeek-V3/R1 training stack, not a clean-room research prototype

Caveats

• The code provides scheduling logic and examples, not a turnkey trainer; you must integrate your own overlapped forward-backward implementation
• Memory footprint per device is strictly higher than simpler schedules (2× parameters, PP+1 activations)
• The comparison formulas assume an even number of pipeline stages

Verdict

Study this if you are building or debugging a pipeline-parallel training framework for large models. Pass if you need a drop-in scheduler that works without custom kernel wiring.