A Transformer that replaced self-attention with autocorrelation

What it does

Autoformer is a long-term time series forecasting model that progressively decomposes inputs into trend and seasonal components while predicting. Instead of point-wise self-attention, it uses an Auto-Correlation mechanism inspired by stochastic process theory to discover period-based dependencies and aggregate information at the series level. The authors report a 38% relative improvement over previous baselines across six benchmarks covering energy, traffic, economics, weather, and disease.

The interesting bit

The model drops position embeddings entirely because the series-wise connection inherently preserves sequential order—an elegant reversal of the usual Transformer recipe. The auto-correlation block is implemented in a batch-normalization-style form to keep memory access friendly, yielding log-linear complexity.

Key highlights

• Deployed operationally for the 2022 Winter Olympics to forecast wind speed and temperature at competition venues
• Extension work published as a Nature Machine Intelligence cover article in 2023
• Now integrated into Hugging Face and the broader Time-Series-Library
• Includes reproducible scripts for six standard benchmarks and comparisons against Informer, Reformer, and vanilla Transformer baselines
• A predict.ipynb notebook provides a walkthrough, though it is written in Chinese

Caveats

• Several promised baselines (LogTrans, N-BEATS) remain unchecked on the todo list
• The quick-start notebook is only available in Chinese, which may limit accessibility for some users

Verdict

Worth a look if you are building or benchmarking long-term forecasters in energy, weather, or traffic domains. Skip it if you need a polished, multilingual MLOps toolkit rather than a research-grade PyTorch implementation.