majumderb/rezero
A PyTorch library implementing ReZero, a technique that initializes neural network layers as identity maps to enable fast convergence in deep Transformers.
Not currently ranked — collecting fresh signals.
star history
This repository provides the ReZero-Transformer implementation, a drop-in replacement for PyTorch’s Transformer. ReZero adds a single learned parameter per layer to initialize arbitrary layers as identity maps, facilitating better gradient flow in deep networks. The technique enables training Transformers over a hundred layers and was shown to converge 56% faster on enwiki8 language modeling tasks. It also applies to ResNets and fully connected networks.
Frequently asked
- What is majumderb/rezero?
- A PyTorch library implementing ReZero, a technique that initializes neural network layers as identity maps to enable fast convergence in deep Transformers.
- Is rezero open source?
- Yes — majumderb/rezero is open source, released under the MIT license.
- What language is rezero written in?
- majumderb/rezero is primarily written in Python.
- How popular is rezero?
- majumderb/rezero has 415 stars on GitHub.
- Where can I find rezero?
- majumderb/rezero is on GitHub at https://github.com/majumderb/rezero.