kyegomez/LongNet
Implementation of LongNet, a transformer architecture using dilated attention to scale to 1 billion token sequences.
Not currently ranked — collecting fresh signals.
star history
This repository provides a plug-and-play implementation of dilated attention from the LongNet research paper. The architecture enables transformers to process extremely long sequences by using dilated attention mechanisms that scale efficiently to 1,000,000,000 tokens. It serves as a direct replacement for standard attention layers in transformer models.
Frequently asked
- What is kyegomez/LongNet?
- Implementation of LongNet, a transformer architecture using dilated attention to scale to 1 billion token sequences.
- Is LongNet open source?
- Yes — kyegomez/LongNet is open source, released under the Apache-2.0 license.
- What language is LongNet written in?
- kyegomez/LongNet is primarily written in Python.
- How popular is LongNet?
- kyegomez/LongNet has 724 stars on GitHub.
- Where can I find LongNet?
- kyegomez/LongNet is on GitHub at https://github.com/kyegomez/LongNet.