Ma-Lab-Berkeley/CRATE
A PyTorch implementation of White-Box Transformers (CRATE) based on sparse rate reduction theory for segmentation and representation learning.
Not currently ranked — collecting fresh signals.
star history
This repository contains the official PyTorch implementation of CRATE (Coding RAte reduction TransformEr), a research framework providing a theoretical understanding of transformer architectures through sparse rate reduction. It reinterprets standard transformer components like multi-head self-attention and feed-forward networks as implementing an information-theoretic objective, providing a white-box alternative to black-box deep learning. The work includes implementations of segmentation and masked autoencoding methods published at top ML venues.
Frequently asked
- What is Ma-Lab-Berkeley/CRATE?
- A PyTorch implementation of White-Box Transformers (CRATE) based on sparse rate reduction theory for segmentation and representation learning.
- Is CRATE open source?
- Yes — Ma-Lab-Berkeley/CRATE is open source, released under the MIT license.
- What language is CRATE written in?
- Ma-Lab-Berkeley/CRATE is primarily written in Python.
- How popular is CRATE?
- Ma-Lab-Berkeley/CRATE has 1.3k stars on GitHub.
- Where can I find CRATE?
- Ma-Lab-Berkeley/CRATE is on GitHub at https://github.com/Ma-Lab-Berkeley/CRATE.