Haiyang-W/TokenFormer
A transformer architecture that tokenizes model parameters and uses attention for both input tokens and learned parameter tokens to enhance scaling flexibility.
Not currently ranked — collecting fresh signals.
star history
TokenFormer replaces fixed linear projection layers with a parameter-efficient attention mechanism that treats model parameters as learnable tokens. This allows the same architecture to scale computations and model parameters jointly through a unified attention framework. The implementation supports training from scratch and provides pre-trained checkpoints for foundation model applications.
Frequently asked
- What is Haiyang-W/TokenFormer?
- A transformer architecture that tokenizes model parameters and uses attention for both input tokens and learned parameter tokens to enhance scaling flexibility.
- Is TokenFormer open source?
- Yes — Haiyang-W/TokenFormer is open source, released under the Apache-2.0 license.
- What language is TokenFormer written in?
- Haiyang-W/TokenFormer is primarily written in Python.
- How popular is TokenFormer?
- Haiyang-W/TokenFormer has 593 stars on GitHub.
- Where can I find TokenFormer?
- Haiyang-W/TokenFormer is on GitHub at https://github.com/Haiyang-W/TokenFormer.