Time series forecasting where dimensions talk to each other cheaply

What it does

Crossformer is a Transformer for multivariate time series forecasting that treats cross-dimension dependency as a first-class citizen. Instead of embedding all variables at a single timestep into one vector, it uses Dimension-Segment-Wise (DSW) chunks—local time segments from each dimension treated as separate tokens. A Hierarchical Encoder-Decoder then aggregates predictions at multiple scales to produce the final forecast. The repo reproduces the ICLR 2023 paper and includes benchmark scripts for ETTh1, Traffic, ECL, and others.

The interesting bit

The Two-Stage Attention (TSA) layer splits the problem into cross-time and cross-dimension stages. Rather than paying an O(D²) bill for naïve multi-head attention across all variable pairs, it funnels information through a small fixed number of router tokens, cutting complexity to O(D). That is the kind of thriftiness you want when D grows.

Key highlights

• DSW embedding treats each dimension’s local time segment as its own token instead of fusing all variables at a single step
• TSA layer uses a router mechanism to share information across dimensions without quadratic blowup
• Hierarchical Encoder-Decoder merges segments at increasing scales and sums forecasts from each level
• Ships with reproducibility scripts for ETTh1, ETTm1, WTH, ECL, ILI, and Traffic datasets
• Supports custom CSV datasets by adjusting dimension, input length, and segment length parameters

Caveats

• The README pins dated dependencies (Python 3.7.10, PyTorch 1.8.1), so expect some environment archaeology on modern hardware
• The save_pred flag explicitly warns it will “cost a lot time and memory for datasets with large D”—a reminder that the router trick fixes complexity, not memory

Verdict

Worth a look if you are forecasting multivariate series where variables actually influence each other and you want something more structured than a flat per-timestep model. Skip it if you need a plug-and-play black box with up-to-date dependencies.