A PyTorch DataLoader that remembers where it left off

What it does

TorchData is Meta’s staging ground for improvements to PyTorch’s core data utilities. Its flagship feature is StatefulDataLoader, a drop-in replacement for the standard PyTorch DataLoader that can save and restore its exact iteration state—including worker buffers and RNG seeds—mid-epoch. It also ships with torchdata.nodes, a library of composable iterators for chaining data preprocessing steps in a streaming fashion.

The interesting bit

The clever part is that StatefulDataLoader is a mechanical drop-in replacement: you swap the class and gain resumability without rewriting your dataset code. The nodes module also insists on being a library of iterators rather than iterables, which nudges you toward a streaming model by default.

Key highlights

• StatefulDataLoader exposes state_dict and load_state_dict for exact mid-epoch resumption.
• Checkpoints capture per-worker state like token buffers and RNG seeds, not just the global iteration counter.
• torchdata.nodes provides composable iterators for building streaming preprocessing pipelines.
• Map-style dataset configurations remain available alongside the streaming model.
• The project is in beta and actively soliciting design feedback from early adopters.

Caveats

• The promised tutorial for torchdata.nodes is still listed as “coming soon,” so the streaming API currently lacks guided documentation.
• The project is explicitly in beta and warns that designs are still being shaped by early feedback.
• Version compatibility is tightly pinned to specific PyTorch releases, so upgrading either package requires consulting the matrix.

Verdict

Worth evaluating if you train long epochs where a mid-crash resume would save serious compute. Pass if your data pipeline is already trivially restartable or you prefer to avoid beta-stage dependencies.