A trainable mashup of sparse and linear attention for diffusion

What it does SLA is a drop-in attention replacement for diffusion transformers that mixes sparse top-k selection with linear feature-map attention. The whole mechanism stays differentiable, so you can fine-tune it end-to-end rather than treating it as a frozen inference hack. The repo ships two versions on separate branches and includes SageSLA, a SageAttention-based fast path for when you just need speed.

The interesting bit Most sparse attention tricks are inference-only patches; SLA is designed to be trained end-to-end, meaning the model learns which tokens to ignore instead of relying on a hand-wired heuristic. It also marries two usually separate acceleration strategies—sparsity and linearization—into one mechanism.

Key highlights

• Fuses sparse top-k selection with linear feature-map attention in a single layer
• Fine-tunable: the sparsity pattern is learned, not hard-coded
• Ships two implementations (main and SLA-1 branches)
• Includes SageSLA, a faster inference variant built on SageAttention
• Targets diffusion transformers and video generation workloads

Caveats

• The README shows API usage but doesn’t quote speedups or accuracy trade-offs in text; you’ll need to check the paper or the effectiveness/efficiency charts for hard numbers
• It is unclear which specific diffusion frameworks are supported out of the box versus requiring manual integration

Verdict Worth a look if you’re training or fine-tuning diffusion transformers and standard full attention has become a compute bottleneck. If you just want a plug-and-play inference drop-in for an existing frozen model, SageSLA might help, but the project’s real pitch is the trainable angle.