Training LLMs to behave, one YAML file at a time

What it does

The Alignment Handbook is a collection of training scripts and YAML recipes that carry a base language model through the entire post-training pipeline—from continued pretraining and supervised fine-tuning to preference alignment with DPO or ORPO. It exists because papers like InstructGPT and Llama 2 demonstrated that preference tuning unlocks significant gains in helpfulness and safety, yet few public resources explain how to collect data, choose metrics, or run the training. Each recipe bundles the hyperparameters, data formatting, and compute configs needed to reproduce specific models like Zephyr 7B, SmolLM3, and StarChat2.

The interesting bit

The project stays intentionally minimal: it is essentially a curated set of scripts and recipes orchestrated through standard Hugging Face tools. The neat trick is treating every training run as a declarative YAML recipe, turning messy alignment experiments into auditable, forkable configs.

Key highlights

• Covers the full alignment stack: continued pretraining, SFT, reward modeling, rejection sampling, DPO, and ORPO.
• Ships reproducible configs for published models, including Zephyr-7b-β, SmolLM3-3B, and StarChat2 15B.
• Supports both full-model distributed training via DeepSpeed ZeRO-3 and parameter-efficient tuning with LoRA/QLoRA.
• Publishes associated datasets and a technical report for transparency.

Verdict

A solid starting point for researchers and engineers who need to replicate published chat models or align their own base models without wiring up a pipeline from scratch. Less useful if you are after a high-level training abstraction or a finished theoretical course on alignment.