lucidrains/linear-attention-transformer
A Transformer variant combining local and global attention mechanisms that scales linearly with sequence length for efficient language modeling.
Not currently ranked — collecting fresh signals.
star history
This repository implements a Transformer architecture with a hybrid attention mechanism combining local (QK^T)V attention with global Q(K^TV) attention for linear time and memory complexity. It includes features like reversible networks, feedforward chunking, and embedding factorization to optimize memory usage. The library is designed for long-sequence language modeling tasks where standard quadratic attention becomes prohibitive.
Frequently asked
- What is lucidrains/linear-attention-transformer?
- A Transformer variant combining local and global attention mechanisms that scales linearly with sequence length for efficient language modeling.
- Is linear-attention-transformer open source?
- Yes — lucidrains/linear-attention-transformer is open source, released under the MIT license.
- What language is linear-attention-transformer written in?
- lucidrains/linear-attention-transformer is primarily written in Python.
- How popular is linear-attention-transformer?
- lucidrains/linear-attention-transformer has 842 stars on GitHub.
- Where can I find linear-attention-transformer?
- lucidrains/linear-attention-transformer is on GitHub at https://github.com/lucidrains/linear-attention-transformer.