An open-source textbook for the algorithm that shapes chatbot personalities

What it does

This repository houses the source for Reinforcement Learning from Human Feedback, a book by Hugging Face’s Nathan Lambert that covers the post-training pipeline used to turn base language models into helpful assistants. The repo includes Markdown chapters, build tooling for HTML/PDF/EPUB, and a growing collection of reference implementations for algorithms like PPO, DPO, GRPO, and rejection sampling.

The interesting bit

Lambert started writing because established techniques like rejection sampling lacked canonical references, while industry practices such as “Character Training” had no open research at all. The book is deliberately positioned as a curated counterweight to “AI slop” — a phrase the author actually uses — with the source code licensed MIT and the text under CC-BY-NC-SA.

Key highlights

• Reference code spans instruction_tuning/, policy_gradients/, reward_models/, direct_alignment/, and rejection_sampling/
• Diagram sources in diagrams/ are reusable for presentations or blog posts, generated via Python scripts, TikZ, or YAML specs
• Build system uses Pandoc templates and Make; outputs HTML, PDF, and EPUB
• Print editions available through Manning and Amazon
• Active Discord community and contributor program with free book copies for substantial contributors

Caveats

• The teach/ directory for slides and courses appears aspirational; no content is visible in the README structure
• Code setup details live in a separate code/README.md, not surfaced here

Verdict

Worth bookmarking if you’re implementing alignment methods or teaching the topic. Skip it if you want a finished, polished reference — the author describes this as an evolving “home base” and acknowledges RLHF itself “will never be a solved problem.”