PyTorch/XLA bridges PyTorch to Google TPUs, but TorchTPU is next

What it does PyTorch/XLA is a Python bridge that feeds PyTorch operations through the XLA deep-learning compiler to run on Cloud TPUs (and CPUs). Training loops need explicit torch_xla.step() wrappers, device transfers to xla, and a final torch_xla.sync() because execution is lazy. It supports single-process, multi-process, and SPMD distribution modes.

The interesting bit The project is living on borrowed time: the README prominently notes that TorchTPU will replace PyTorch/XLA once it goes public, following an earlier RFC for a more native PyTorch-on-TPU direction. That makes this less a permanent platform and more a functional off-ramp.

Key highlights

• Compiles PyTorch graphs to XLA for execution on Cloud TPUs; free single-TPU trials are available via Kaggle notebooks
• Distributed training uses PJRT, with DDP supported through an xla:// process-group backend that infers rank and world size automatically
• Switching to C++11 ABI wheels/docker images improves lazy-tensor tracing performance; the README cites Mixtral 8x7B going from 33% to 39% MFU on a v5p-256
• Documentation covers SPMD, FSDP, Dynamo, quantization, recompilation, and custom Pallas kernels
• Reference LLM and diffusion-model implementations live in the separate AI-Hypercomputer/tpu-recipes repository

Caveats

• Multi-processing mode and SPMD are explicitly incompatible; you have to pick one distribution strategy
• The README warns that TorchTPU is queued to replace this project, so new work may need migration sooner rather than later
• It is not a transparent drop-in: you must move tensors and models to the xla device and manage synchronization manually

Verdict Worth a look if you are already committed to Google Cloud TPUs and need PyTorch support right now. Skip it if you are on NVIDIA hardware, allergic to migration work, or waiting for the promised native TorchTPU experience.