Stretching LLaMA to 100k tokens without stretching your GPU budget

What it does LongLoRA is a fine-tuning recipe and model zoo for stretching LLaMA-family context windows from 4k up to 100k tokens. It provides training scripts, evaluation code, and ready-made weights from 7B to 70B parameters, alongside the LongAlpaca-12k instruction dataset—9k long Q&A pairs blended with 3k short ones to keep brief-task performance intact.

The interesting bit The core mechanism is “shifted short attention”: during training the model processes long sequences in chunks, but half the attention heads are shifted to preserve cross-group relationships. At inference the shift is discarded and standard attention takes over, so serving infrastructure doesn’t change. It is compatible with Flash Attention and can be layered with QLoRA for further memory savings.

Key highlights

• Released checkpoints cover 7B through 70B models and context lengths from 8k to 100k; the authors note this includes the first open long-context 70B model.
• The shifted short attention implementation is simple and works with Flash Attention; it is only needed during training, not inference.
• Supports GPT-NeoX architectures and integrates with StreamingLLM for extended multi-round dialogue.
• Code is Apache 2.0, but the pre-trained weights and LongAlpaca dataset are under CC BY-NC 4.0.

Caveats

• Weights and dataset are under CC BY-NC 4.0, so commercial use is off the table.
• Downloading checkpoints requires a Hugging Face account and agreeing to Meta’s Llama license.

Verdict Worth a look for researchers and experimenters who want long-context LLaMA models without rewriting the attention kernel. Avoid if you need a commercially unrestricted drop-in model, since the weights and dataset are non-commercial.