Wrap, don't fork: distributed training for Hugging Face models

What it does

Halo is a PyTorch training framework for large language and multimodal models. It handles pre-training, supervised fine-tuning, preference optimization, and asynchronous multi-turn RL on setups ranging from a single GPU to multi-node clusters. The codebase trains standard Hugging Face models directly—checkpoints remain ordinary SafeTensors and load with from_pretrained—while applying Expert, Context, Tensor, and FSDP parallelism to the existing model architecture rather than forking it into a separate distributed implementation.

The interesting bit

Instead of the usual Megatron-style rewrite, Halo wraps MoE blocks, attention layers, and weights in place with EP, CP, TP, and ETP. On the RL side, it keeps the trainer in plain Transformers and pushes updated weights to a vLLM or SGLang rollout server over native NCCL, avoiding veRL dependencies. The project even maintains separate documentation tracks for humans and coding agents, complete with discoverable skills for Claude Code and Codex.

Key highlights

• Native Hugging Face compatibility: new model families reportedly need under 140 lines of integration code, and checkpoints never leave the standard ecosystem.
• In-place parallelism: EP, CP, TP, and ETP are layered onto existing modules via PyTorch DTensor and FSDP2, with DeepEP handling MoE all-to-all traffic.
• Asynchronous RL: multi-turn rollouts run as Ray actors against vLLM or SGLang, overlapping training through a prefetch queue while the trainer stays vanilla Transformers.
• Integrated kernels: ships with FlashAttention 4, Liger fused ops, Grouped GEMM, padding-free packing, and an AdamWBF16 optimizer that cuts optimizer state from 12 to 6 bytes per parameter.
• Self-reported speedups: on 8× B300, Halo claims up to roughly 2.8× the throughput of stock TRL with ZeRO-3 sharding, plus lower peak memory.

Caveats

• Expert Tensor Parallelism (ETP) is explicitly marked experimental in the docs.
• Docker images are split by GPU architecture (Blackwell vs. Hopper) with no latest tag, so picking the wrong image is a footgun.
• The project just shipped its first public release, so real-world mileage is still accumulating.

Verdict

Worth a look if you are post-training or pre-training Hugging Face models and want to keep the standard from_pretrained workflow at scale. Less compelling if you are already locked into Megatron-LM or need a training stack with years of production baggage.