lucidrains/recurrent-memory-transformer-pytorch
A PyTorch implementation of the Recurrent Memory Transformer architecture for processing long sequences using memory tokens.
Not currently ranked — collecting fresh signals.
star history
This repository implements the Recurrent Memory Transformer (RMT) paper in PyTorch, introducing memory tokens that allow transformers to handle very long contexts by compressing information across segments. The architecture passes memory embeddings between segments, enabling information retention across sequences of arbitrary length. It includes support for flash attention and is designed for autoregressive sequence modeling tasks.
Frequently asked
- What is lucidrains/recurrent-memory-transformer-pytorch?
- A PyTorch implementation of the Recurrent Memory Transformer architecture for processing long sequences using memory tokens.
- Is recurrent-memory-transformer-pytorch open source?
- Yes — lucidrains/recurrent-memory-transformer-pytorch is open source, released under the MIT license.
- What language is recurrent-memory-transformer-pytorch written in?
- lucidrains/recurrent-memory-transformer-pytorch is primarily written in Python.
- How popular is recurrent-memory-transformer-pytorch?
- lucidrains/recurrent-memory-transformer-pytorch has 424 stars on GitHub.
- Where can I find recurrent-memory-transformer-pytorch?
- lucidrains/recurrent-memory-transformer-pytorch is on GitHub at https://github.com/lucidrains/recurrent-memory-transformer-pytorch.