An Open LLM with a Training Diary and an Expiration Date

What it does OLMo is AI2’s training and inference codebase for its OLMo-2 model family, spanning 1B to 32B parameters. It publishes not just final weights but every training config, intermediate checkpoint, and data provenance log needed to reproduce the runs from scratch. The pipeline follows a two-stage recipe: first pretraining on 4–5 trillion mostly web-based tokens, then continuing on smaller, curated high-quality datasets.

The interesting bit The project treats model training like a lab notebook rather than a product launch. For the 7B and 13B variants, the team trains multiple stage-two runs with different data-shuffling seeds and literally averages—or “soups”—the weights together, publishing the WandB curves and .csv.gz provenance files alongside every checkpoint.

Key highlights

• Full reproducibility stack: configs, checkpoints saved at least every 1000 steps, WandB logs, and compressed CSV data provenance files
• Two-stage curriculum: mass pretraining on web data followed by targeted refinement on high-quality mixes
• Model “souping”: final 7B and 13B checkpoints are weight averages of independently trained seed variants
• Dual-format releases: native OLMo format and Hugging Face format for all sizes
• Apple Silicon support for small-scale experiments, though serious training expects multi-GPU clusters

Caveats

• Deprecated: The README prominently warns this repository is out of date and no longer active; newer releases and the 32B trainer live in allenai/OLMo-core.
• Streaming by default: Training configs point to HTTP data streams, and the authors explicitly recommend switching to local file paths for any large-scale reproduction.
• Inference-light docs: The README is overwhelmingly focused on pretraining tables and checkpoints; evaluation and inference details are comparatively sparse.

Verdict Researchers who want to replicate or dissect LLM pretraining will find the published configs and checkpoints essential reading. Everyone else should follow the README’s own advice and head to the newer OLMo-core repository.