This repository provides a clean implementation of the PaLM Transformer architecture in under 200 lines of code. It replicates the key components of Google’s Pathways Language Model including SwiGLU activation, parallel attention layers, and RoPE embeddings. The implementation supports configurable model dimensions up to the 540B parameter scale described in the original paper and includes training scripts for language modeling benchmarks like Enwik8.