lucidrains/reformer-pytorch
A PyTorch implementation of the Reformer efficient Transformer model with LSH attention, reversible layers, and chunking for memory-efficient training.
Not currently ranked — collecting fresh signals.
star history
This repository provides a PyTorch implementation of the Reformer architecture, an efficient Transformer variant designed to reduce memory consumption during training and inference. It includes key innovations from the original paper: locality-sensitive hashing (LSH) attention to handle long sequences, reversible residual networks for memory-efficient backpropagation, and chunking techniques. The implementation supports auto-regressive language modeling tasks and has been validated on the enwik8 dataset with sequences up to 81k tokens using half precision.
Frequently asked
- What is lucidrains/reformer-pytorch?
- A PyTorch implementation of the Reformer efficient Transformer model with LSH attention, reversible layers, and chunking for memory-efficient training.
- Is reformer-pytorch open source?
- Yes — lucidrains/reformer-pytorch is open source, released under the MIT license.
- What language is reformer-pytorch written in?
- lucidrains/reformer-pytorch is primarily written in Python.
- How popular is reformer-pytorch?
- lucidrains/reformer-pytorch has 2.2k stars on GitHub.
- Where can I find reformer-pytorch?
- lucidrains/reformer-pytorch is on GitHub at https://github.com/lucidrains/reformer-pytorch.