datamllab/LongLM
Self-Extend extends LLM context windows without tuning by adding binary attention stratification based on relative distance.
Not currently ranked — collecting fresh signals.
star history
This repository implements Self-Extend, a technique that enables large language models to handle longer context windows without requiring fine-tuning. It works by grouping attention into local and global buckets based on relative token distance, allowing models to reason beyond their native attention span. The implementation includes optimized versions using FlashAttention and Triton, with support for various LLMs including Llama, Qwen, and Gemma.
Frequently asked
- What is datamllab/LongLM?
- Self-Extend extends LLM context windows without tuning by adding binary attention stratification based on relative distance.
- Is LongLM open source?
- Yes — datamllab/LongLM is open source, released under the MIT license.
- What language is LongLM written in?
- datamllab/LongLM is primarily written in Python.
- How popular is LongLM?
- datamllab/LongLM has 668 stars on GitHub.
- Where can I find LongLM?
- datamllab/LongLM is on GitHub at https://github.com/datamllab/LongLM.