Dynamic CPU-GPU paging for 100B+ model training

What it does PatrickStar is a PyTorch training system that stretches GPU memory by offloading model parameters to CPU RAM and only keeping active computation chunks on the card. The team reports training an 18-billion-parameter model on eight Tesla V100s, a 68-billion-parameter model on eight A100s, and even running a 175-billion-parameter GPT-3-class model on just 32 GPUs.

The interesting bit Most heterogeneous-training frameworks split data statically between CPU and GPU, which leaves GPU memory underused when activations fluctuate. PatrickStar instead uses a chunk-based memory manager that dynamically migrates parameters based on what is currently computing. The authors claim this lets them fit models two to six times larger than DeepSpeed on identical hardware, with better throughput for same-size models as well.

Key highlights

• Claims over 2× the model scale of DeepSpeed on 8×V100 nodes (18B parameters).
• Reports a 68B model on 8×A100 with 1 TB CPU memory, roughly 6× larger than DeepSpeed v0.5.7.
• Says it is the first work to run a 175B GPT-3-class model on a 32-GPU cluster, compared with the original 10,000-V100 setup.
• Uses a DeepSpeed-compatible JSON config and an initialize_engine wrapper, so existing PyTorch projects can migrate without rewriting training loops.
• Ships with self-contained benchmark scripts that use randomly generated data, letting you test memory scaling without curating a dataset.

Verdict Worth a look if you are bumping into OOM errors on your current cluster and want to squeeze in larger models before buying more silicon. Skip it if your models already fit comfortably in VRAM.