lucidrains/PaLM-pytorch
A PyTorch implementation of the PaLM (Pathways Language Model) Transformer architecture for scaled language modeling.
Not currently ranked — collecting fresh signals.
star history
This repository provides a clean implementation of the PaLM Transformer architecture in under 200 lines of code. It replicates the key components of Google’s Pathways Language Model including SwiGLU activation, parallel attention layers, and RoPE embeddings. The implementation supports configurable model dimensions up to the 540B parameter scale described in the original paper and includes training scripts for language modeling benchmarks like Enwik8.
Frequently asked
- What is lucidrains/PaLM-pytorch?
- A PyTorch implementation of the PaLM (Pathways Language Model) Transformer architecture for scaled language modeling.
- Is PaLM-pytorch open source?
- Yes — lucidrains/PaLM-pytorch is open source, released under the MIT license.
- What language is PaLM-pytorch written in?
- lucidrains/PaLM-pytorch is primarily written in Python.
- How popular is PaLM-pytorch?
- lucidrains/PaLM-pytorch has 825 stars on GitHub.
- Where can I find PaLM-pytorch?
- lucidrains/PaLM-pytorch is on GitHub at https://github.com/lucidrains/PaLM-pytorch.