A Transformer that skips the lazy attention queries

What it does Informer is a Transformer architecture built for long-sequence time-series forecasting. It swaps standard self-attention for ProbSparse Attention, which keeps only the “active” queries from the head of the attention-score distribution and drops the “lazy” ones in the tail. This repo is the official PyTorch implementation of the AAAI 2021 Best Paper, bundled with training scripts, Colab notebooks, and benchmark datasets for electricity, weather, and ETT.

The interesting bit The authors justify sparsifying queries instead of keys by arguing that top queries with full keys preserve the input’s leading components, whereas top keys would merely sum the long-tail noise and wreck the representation. It is an asymmetric trick that sounds obvious once you read the footnote.

Key highlights

• Official AAAI'21 Best Paper code supporting univariate and multivariate forecasting modes (M, S, MS).
• ProbSparse Attention plus a generative decoder; an informerstack variant is included, though informerlight remains TBD.
• Hyperparameter shell scripts and updated result tables after a community-reported data-scaling correction.
• Google Colab notebooks for training, prediction, and visualization.
• CC BY-NC-SA license, which keeps it firmly in the research and non-commercial lane.

Caveats

• Dependency stack is locked to Python 3.6 and PyTorch 1.8.0; the FAQ warns of Conv1d padding mismatches on newer torch versions.
• informerlight is listed as TBD in the model options.
• The non-commercial CC BY-NC-SA license rules out production usage without separate permission.

Verdict Use it as a strong research baseline for long-horizon forecasting if you can tolerate a frozen dependency stack and a non-commercial license. Pass if you need a modern, production-ready library with commercial clearance.