Before GPT-NeoX, there was mesh-tensorflow and hope

What it does

GPT-Neo is a TensorFlow implementation of GPT-2/GPT-3-style transformers that uses mesh-tensorflow to shard model weights and data across TPU pods. It trains and runs inference on configurations up to a few billion parameters, with pretrained 1.3B and 2.7B checkpoints released. The codebase also supports local attention, linear attention, mixture-of-experts layers, and axial positional embeddings.

The interesting bit

It is essentially a historical artifact: development froze in August 2021 because mesh-tensorflow buckles at 200B+ parameters and the team acquired enough GPUs to justify rewriting everything as GPT-NeoX. That makes the repo a snapshot of the TPU-first era of open-source large-model training.

Key highlights

• Archived since August 2021; preserved for anyone still maintaining legacy TPU pipelines.
• Ships with pretrained 1.3B and 2.7B models trained on The Pile, though the authors now point users to HuggingFace for inference.
• Adds experimental attention variants (local, linear) and Mixture-of-Experts not present in the original GPT-3.
• Evaluation benchmarks show the 2.7B model generally trailing GPT-3 2.7B but beating GPT-2 1.5B on several tasks.
• Officially TPU-native; GPU support exists but users report mesh-tensorflow struggling to recognize GPUs.

Caveats

• No longer maintained; active development moved to GPT-NeoX.
• Inefficient beyond ~200B parameters, so it never reached the true GPT-3 scale.
• GPU training is unofficial and reportedly finicky.

Verdict

Worth a look if you are studying the archaeology of open-source LLMs or maintaining an existing TPU fleet. Everyone else should use GPT-NeoX or the HuggingFace integration instead.