kingoflolz/mesh-transformer-jax
A model-parallel transformer library in JAX and Haiku designed to scale up to 40B parameters on TPUs, hosting the GPT-J-6B language model.
Not currently ranked — collecting fresh signals.
star history
This repository provides a haiku library leveraging JAX’s xmap/pjit operators for transformer model parallelism, implementing a scheme similar to Megatron-LM optimized for TPU meshes with optional ZeRo-style sharding. It includes GPT-J-6B, a 6 billion parameter autoregressive language model trained on The Pile, along with support for fine-tuning and partial checkpoint training. The codebase is designed for scalability up to approximately 40B parameters.
Frequently asked
- What is kingoflolz/mesh-transformer-jax?
- A model-parallel transformer library in JAX and Haiku designed to scale up to 40B parameters on TPUs, hosting the GPT-J-6B language model.
- Is mesh-transformer-jax open source?
- Yes — kingoflolz/mesh-transformer-jax is open source, released under the Apache-2.0 license.
- What language is mesh-transformer-jax written in?
- kingoflolz/mesh-transformer-jax is primarily written in Python.
- How popular is mesh-transformer-jax?
- kingoflolz/mesh-transformer-jax has 6.4k stars on GitHub.
- Where can I find mesh-transformer-jax?
- kingoflolz/mesh-transformer-jax is on GitHub at https://github.com/kingoflolz/mesh-transformer-jax.