tczhangzhi/pytorch-distributed
A quickstart and benchmark collection for distributed PyTorch training using DataParallel, Distributed, Apex, and Horovod.
Not currently ranked — collecting fresh signals.
star history
This repository provides code examples and benchmarks for distributed training in PyTorch across single-machine multi-GPU scenarios. It covers implementations using nn.DataParallel, torch.distributed, torch.multiprocessing, NVIDIA Apex, and Horovod, along with SLURM cluster configurations. The benchmarks compare training speed on ImageNet with Tesla V100 GPUs, finding Apex and Horovod achieve similar performance while DataParallel is notably slower.
Frequently asked
- What is tczhangzhi/pytorch-distributed?
- A quickstart and benchmark collection for distributed PyTorch training using DataParallel, Distributed, Apex, and Horovod.
- Is pytorch-distributed open source?
- Yes — tczhangzhi/pytorch-distributed is open source, released under the MIT license.
- What language is pytorch-distributed written in?
- tczhangzhi/pytorch-distributed is primarily written in Python.
- How popular is pytorch-distributed?
- tczhangzhi/pytorch-distributed has 1.7k stars on GitHub.
- Where can I find pytorch-distributed?
- tczhangzhi/pytorch-distributed is on GitHub at https://github.com/tczhangzhi/pytorch-distributed.