Jamie-Stirling/RetNet
A minimal pure PyTorch implementation of RetNet, an alternative neural architecture to Transformers for large language models.
Not currently ranked — collecting fresh signals.
star history
This repository provides a PyTorch implementation of the Retentive Network architecture described in the paper ‘Retentive Network: A Successor to Transformer for Large Language Models’. It implements single-scale and multi-scale retention mechanisms across parallel, recurrent, and chunkwise computation paradigms. The codebase includes multi-layer networks with feed-forward layers and layer normalization, plus a causal language model built on the retentive architecture. It prioritizes code correctness and readability over optimization.
Frequently asked
- What is Jamie-Stirling/RetNet?
- A minimal pure PyTorch implementation of RetNet, an alternative neural architecture to Transformers for large language models.
- Is RetNet open source?
- Yes — Jamie-Stirling/RetNet is open source, released under the MIT license.
- What language is RetNet written in?
- Jamie-Stirling/RetNet is primarily written in Python.
- How popular is RetNet?
- Jamie-Stirling/RetNet has 1.2k stars on GitHub.
- Where can I find RetNet?
- Jamie-Stirling/RetNet is on GitHub at https://github.com/Jamie-Stirling/RetNet.