abertsch72/unlimiformer
An implementation of retrieval-based attention augmentation for encoder-decoder transformers to handle unlimited-length inputs, supporting Llama-2 and its derivatives.
Not currently ranked — collecting fresh signals.
star history
Unlimiformer augments pretrained encoder-decoder models with retrieval-based attention, allowing unlimited length inputs without changing the mathematical definition of attention. It can improve already-trained models or be used during training for best results. The implementation supports arbitrary pretrained models including Llama-2, enabling use cases like summarizing entire books.
Frequently asked
- What is abertsch72/unlimiformer?
- An implementation of retrieval-based attention augmentation for encoder-decoder transformers to handle unlimited-length inputs, supporting Llama-2 and its derivatives.
- Is unlimiformer open source?
- Yes — abertsch72/unlimiformer is open source, released under the MIT license.
- What language is unlimiformer written in?
- abertsch72/unlimiformer is primarily written in Python.
- How popular is unlimiformer?
- abertsch72/unlimiformer has 1.1k stars on GitHub.
- Where can I find unlimiformer?
- abertsch72/unlimiformer is on GitHub at https://github.com/abertsch72/unlimiformer.