A partial Transformer replication, honestly labeled

What it does

Reproduces the core synthetic-dataset experiment from a 2019 NeurIPS paper on using Transformers for time-series forecasting. The author adds local convolutional preprocessing to help the self-attention mechanism capture nearby temporal structure, then trains with teacher forcing and visualizes where the attention heads focus.

The interesting bit

The README’s all-caps disclaimer is the real feature: “THIS IS NOT THE PAPERS CODE. THIS DOES NOT HAVE SPARSITY.” Most repos quietly omit what they skipped; this one broadcasts it. The attention visualization showing how layers attend to specific signal regions for predicting timestep t=t0+24-1 is the payoff for the synthetic setup.

Key highlights

• Matches paper results on the synthetic dataset (shown in an Rp comparison table)
• Uses synthetic data with known structure to isolate whether the model learns local dependencies
• Attention-weight heatmaps reveal which time steps the model deems relevant
• Jupyter Notebook implementation, not a production framework

Caveats

• Deliberately omits the paper’s sparse attention mechanism, a central contribution
• Teacher-forced training only; no autoregressive rollout evaluation shown
• Scope is narrow: one synthetic benchmark, not real-world series

Verdict

Worth a look if you want a readable, warts-and-all walkthrough of Transformer attention on toy time-series data. Skip it if you need the full sparse-attention method or anything beyond notebook-scale experimentation.