Open-source ChatGPT, minus the millions and the weights

What it does This repo lays out the full three-stage RLHF pipeline in PyTorch: an autoregressive PaLM transformer, a binned RewardModel that scores completions, and a PPO-driven RLHFTrainer that refines the policy against those scores. It is explicitly a skeleton—there are no pretrained weights, so out of the box it generates random tokens.

The interesting bit The author is admirably blunt that this is “just the ship and overall map,” and even flags Direct Preference Optimization as a likely successor that could replace the whole reward-model-and-PPO complexity with a few lines of cross-entropy. That honesty is refreshing in a field crowded with repos that imply inference-ready magic.

Key highlights

• Implements the full RLHF loop: base PaLM model, RewardModel, and RLHFTrainer.
• Reward model supports both full finetuning and LoRA adapters, acknowledging the original paper struggled with overfitting here.
• Integrated Flash Attention from PyTorch 2.0 for memory-efficient training.
• Explicitly disclaims containing a trained model; it is architecture and training logic only.

Caveats

• No trained weights are provided, and the README notes you still need “millions of dollars of compute + data” plus expert tuning to produce anything useful.
• Several trainer conveniences remain unfinished, including Hugging Face accelerate integration, variable-length prompt sampling, and a human-feedback collection UI.

Verdict Worth a look if you are researching RLHF mechanics or building a custom training stack from first principles. If you need an off-the-shelf conversational model to deploy today, this is explicitly not it.