This repository provides a pre-training implementation of Google’s LaMDA model architecture in PyTorch. It replicates key architectural features including T5 relative positional bias, gated GELU activations, and a decoder-only autoregressive design. The project incorporates reinforcement learning from human feedback (RLHF) to enable training similar to ChatGPT, alongside HuggingFace dataset integration for efficient data loading and a GPT-2 compatible tokenizer.