This Transformer Detects Anomalies by Measuring Self-Attention

What it does

The repo is the official PyTorch release of the ICLR 2022 Spotlight paper Anomaly Transformer. It tackles unsupervised point-wise anomaly detection in time series by deriving a detection criterion directly from the model’s own attention maps. Instead of relying on reconstruction error or forecast deviation, it flags outliers using an internal metric called association discrepancy, amplified through a minimax training strategy.

The interesting bit

The core insight is that anomalous points pay attention differently: they associate strongly with nearby timestamps but weakly with the broader series, whereas normal points do the opposite. The Anomaly-Attention module explicitly computes both of these association types, and the minimax objective cranks up the volume on that gap so the outliers become unmistakable.

Key highlights

• Introduces Association Discrepancy as an intrinsic, model-native detection criterion rather than a post-hoc reconstruction loss.
• Implements a custom Anomaly-Attention mechanism inside the Transformer to compute the local-vs-global attention divergence.
• Uses a minimax strategy specifically to magnify the distinguishability between normal and abnormal association patterns.
• Ships with pre-processed benchmark datasets (SMD, MSL, SMAP, PSM) and scripts to reproduce comparisons against 15 baselines including THOC and InterFusion.
• Reports SOTA results on the provided benchmarks.

Caveats

• Evaluation depends on a specific “adjustment operation” that has generated enough questions to warrant a dedicated issue and author clarification.
• The SWaT dataset is not bundled; obtaining it requires a separate application through its official channel.
• Environment setup has needed community patches to resolve dependency issues.

Verdict

Worth a look if you are building or benchmarking unsupervised time-series anomaly detectors and want a method that treats attention patterns as the primary evidence. Skip it if you need a turnkey production pipeline with standardized evaluation and fully included datasets.