Distributed training that skips the parameter-server mess

What it does

Horovod is a distributed training framework for TensorFlow, Keras, PyTorch, and MXNet. It wraps your existing optimizer to average gradients across workers using allreduce or allgather, and handles the boilerplate of multi-process initialization and GPU pinning. The pitch is that you write for one GPU, then scale to hundreds without touching the model code again.

The interesting bit

Instead of rebuilding around parameter servers, Horovod leans on old-school MPI concepts—rank, broadcast, allreduce—which turn out to be a surprisingly good fit for gradient synchronization. It also bundles Facebook’s Gloo backend, so you can dodge an MPI installation entirely if you prefer.

Key highlights

• Supports TensorFlow, Keras, PyTorch, and Apache MXNet from a single API
• Benchmarked at 512 GPUs across 128 servers with claimed 90% scaling efficiency on ResNet-101 and Inception V3
• Same script runs on one GPU, multiple GPUs, or multiple hosts with no code changes
• Can use either MPI or the built-in Gloo collective communications library
• Plays nice with mpi4py and runs on Kubernetes, Spark, Ray, and HPC clusters

Caveats

• The README admits that installing MPI and NCCL is “an extra hassle,” though it only needs to happen once per infrastructure team
• Compiler requirements vary by framework version—TensorFlow 2.10+ needs g++8 or above—so the build matrix can get picky

Verdict

Worth a look if you’re already training in PyTorch or TensorFlow and need to scale past a single node without re-architecting. Probably overkill if you’re still experimenting on a single GPU or running small models that don’t need synchronous distributed training.