DeepSeek’s Native Sparse Attention, rebuilt in PyTorch

What it does

This repo implements the sparse attention pattern from DeepSeek’s Native Sparse Attention paper. It provides a SparseAttention module that replaces standard self-attention with a mix of sliding windows, compressed blocks, and selected blocks, plus an Enwik8 training script to check that the setup actually learns.

The interesting bit

The author credits PyTorch Flex Attention and Triton for enabling rapid prototyping without raw CUDA hacking. He also notes this is his final open-source release while at Meta, giving the repo a slightly valedictory air.

Key highlights

• Reproduces the attention pattern from DeepSeek’s arXiv:2502.11089 paper.
• Credits PyTorch Flex Attention for rapid prototyping without hand-written kernels.
• Exposes tunable block sizes via compress_block_size, selection_block_size, and num_selected_blocks.
• Includes an Enwik8 language modeling example to verify training works.
• Community fixes have already corrected compression hyperparameters and added a memory access guard.

Caveats

• The README is essentially a quickstart: it lists hyperparameters but does not explain how the blocks interact, so you will need the paper for intuition.
• The author describes the project as his last open-source release under Meta, with no further roadmap promised.

Verdict

Worth cloning if you want a readable PyTorch reference for sparse attention research. Skip it if you need a production-grade kernel or a self-contained tutorial on the algorithm.