lucidrains/memorizing-transformers-pytorch
A PyTorch implementation of Memorizing Transformers, a transformer architecture augmented with approximate nearest neighbor memory retrieval.
Not currently ranked — collecting fresh signals.
star history
This repository provides a PyTorch implementation of the Memorizing Transformers paper from ICLR 2022. The model augments standard transformer attention with an external memory system that uses approximate nearest neighbors for retrieval. During inference, the model retrieves relevant past tokens from a memory store to enhance context understanding. The implementation uses cosine similarity attention with learned temperature for the KNN attention layer, and supports hybrid attention across local and distant contexts.
Frequently asked
- What is lucidrains/memorizing-transformers-pytorch?
- A PyTorch implementation of Memorizing Transformers, a transformer architecture augmented with approximate nearest neighbor memory retrieval.
- Is memorizing-transformers-pytorch open source?
- Yes — lucidrains/memorizing-transformers-pytorch is open source, released under the MIT license.
- What language is memorizing-transformers-pytorch written in?
- lucidrains/memorizing-transformers-pytorch is primarily written in Python.
- How popular is memorizing-transformers-pytorch?
- lucidrains/memorizing-transformers-pytorch has 646 stars on GitHub.
- Where can I find memorizing-transformers-pytorch?
- lucidrains/memorizing-transformers-pytorch is on GitHub at https://github.com/lucidrains/memorizing-transformers-pytorch.