haoliuhl/ringattention
A distributed attention mechanism for training large language models with context lengths up to tens of millions of tokens.
Not currently ranked — collecting fresh signals.
star history
This repository provides a JAX implementation of Ring Attention with Blockwise Parallel Transformers, enabling near-infinite context training for large language models. The approach distributes attention and feedforward computation across multiple devices while overlapping communication with computation, allowing training without additional overhead. It handles blockwise computing of attention and feedforward networks to efficiently scale to very long sequences.
Frequently asked
- What is haoliuhl/ringattention?
- A distributed attention mechanism for training large language models with context lengths up to tens of millions of tokens.
- Is ringattention open source?
- Yes — haoliuhl/ringattention is open source, released under the Apache-2.0 license.
- What language is ringattention written in?
- haoliuhl/ringattention is primarily written in Python.
- How popular is ringattention?
- haoliuhl/ringattention has 773 stars on GitHub.
- Where can I find ringattention?
- haoliuhl/ringattention is on GitHub at https://github.com/haoliuhl/ringattention.