Teaching Vision Transformers to Look Where It Matters

What it does

DAT and DAT++ are backbone Vision Transformers that trade both global self-attention and fixed window constraints for a learned, sparse sampling scheme. A lightweight offset network shifts the reference points for key and value extraction, letting each query attend to image regions that actually matter rather than every token or rigid grid cell. This repository hosts ImageNet-1K classification code and pretrained weights; detection and segmentation are handled in separate repos.

The interesting bit

The trick is borrowing deformable convolutions’ offset mechanism but applying it to self-attention, which sidesteps the quadratic cost of dense global attention without forcing the rigid grid of shifted windows. The receptive field grows where the data demands it rather than where the architecture dictates.

Key highlights

• CVPR 2022 Best Paper Finalist; extended DAT++ version also available
• ImageNet-1K accuracy up to 85.9% for DAT-B++ at 384×384 resolution
• Classification focus here; object detection and semantic segmentation maintained in separate repositories (DAT-Detection, DAT-Segmentation)
• Built atop the Swin Transformer codebase
• Pretrained weights provided for Tiny, Small, and Base model sizes

Caveats

• ImageNet-22K pretraining for larger models and native CUDA/CUTLASS acceleration are still on the roadmap, not yet shipped
• Detection and segmentation training pipelines are not integrated here; this repo is strictly classification

Verdict

Worth a look if you are researching attention efficiency in vision backbones or need a drop-in alternative to Swin/ViT for image classification. Skip it if you want an end-to-end detection or segmentation training pipeline in one repo.