Self-attention that behaves like a convolution kernel

What it does

NATTEN is an infrastructure library for multi-dimensional sparse attention, built around Neighborhood Attention (NA). Instead of flattening images or volumes into 1-D sequences and computing dense global attention, it keeps the spatial layout and restricts each token to attend only to its local neighborhood, much like a convolution. It ships fused training and inference kernels for PyTorch, covering NVIDIA architectures from Maxwell through Blackwell.

The interesting bit

The library treats attention as a spatial operation with hyperparameters you’d expect from a convolution layer—kernel_size, stride, and dilation—and supports causal masking per axis. The latest release adds native Hopper and Blackwell kernels that the authors say deliver speedups proportional to FLOP reductions over cuDNN and Flash Attention 3, plus up to a 1.47× inference boost on Ampere in fully block-sparse cases.

Key highlights

• Supports 2-D and 3-D token layouts natively, so feature maps don’t need to be flattened into sequences.
• Fused Neighborhood Attention (FNA) and FMHA kernels for NVIDIA GPUs from Maxwell (SM50) through Blackwell (SM100/SM103).
• Per-dimension control over kernel_size, stride, dilation, and causal masking.
• Includes a profiling toolkit for analyzing kernel performance.
• Version 0.21.6 introduces Hopper and Blackwell native kernels and faster Ampere inference for block-sparse patterns.

Caveats

• Speedup claims are explicitly tied to FLOP reduction; dense or non-sparse configurations may not see comparable gains.
• Requires PyTorch 2.7 or newer, leaving older PyTorch versions behind.
• The entire kernel catalog is NVIDIA-specific; no AMD, Intel, or Apple Silicon backends are mentioned.

Verdict

Worth a look if you’re building vision transformers or video models and want to replace dense attention with something more spatially grounded. If you’re on non-NVIDIA hardware or need sequence-level attention over long 1-D text, this is not your library.