Stretching LLaMA to 128K contexts the reproducible way

What it does

YaRN is a research artifact for an ICLR 2024 paper on extending LLM context windows. The repository bundles training code, evaluation scripts, and tokenized datasets so others can reproduce the published results. It also distributes ready-made fine-tuned checkpoints that push Llama 2, Mistral 7B, and SOLAR to between 32K and 128K tokens.

The interesting bit

Rather than training long-context models from scratch, the project fine-tunes existing weights to handle longer sequences efficiently. It publishes the exact checkpoints, training data, and DeepSpeed configurations, which is unusually thorough for reproducibility in this space.

Key highlights

• ICLR 2024 paper with fully open training code and data
• Published fine-tunes: Llama 2 (7B/13B/70B), Mistral 7B, and SOLAR 10.7B at 32K–128K context
• Tokenized training datasets derived from PG-19 and Long-Data-Collections hosted on Hugging Face
• Evaluation pipeline built on EleutherAI’s lm-evaluation-harness
• Uses DeepSpeed ZeRO-3 for distributed training

Caveats

• The README never explains the YaRN algorithm itself; understanding the method requires reading the paper
• The repository is a reproduction bundle tied to specific model families, not a general-purpose context-extension library
• A “v2” of the paper is mentioned but the differences from v1 are not described

Verdict

Researchers who need long-context open models and want to verify or extend published methods will find this valuable. If you’re looking for a plug-and-play library to extend arbitrary models without fine-tuning, this isn’t it.