Llama-2, but Chinese, 64K-context patient, and RLHF-aligned

What it does

Chinese-LLaMA-Alpaca-2 takes Meta’s Llama-2 and incrementally pre-trains it on 120 GB of Chinese text using a redesigned 55,296-token vocabulary shared across base and chat variants. It releases both base models (Chinese-LLaMA-2) and instruction-tuned chat models (Chinese-Alpaca-2) at 1.3B, 7B, and 13B sizes, plus long-context versions that stretch to 16K or 64K tokens and RLHF-tuned variants meant to improve correct values. Training and fine-tuning scripts are included for replication.

The interesting bit

The vocabulary expansion is the quiet workhorse: instead of forcing Chinese through Llama-2’s English-centric tokenizer, the team unified the base and chat vocabularies to improve encoding efficiency. For context stretching, they trained dedicated long-context models with Position Interpolation (16K, extendable to roughly 24K–32K via NTK) and YaRN (64K), rather than relying solely on inference-time tricks.

Key highlights

• Redesigned 55,296-token Chinese vocabulary, shared between base and chat models to avoid cross-model tokenization mismatches.
• Standard 4K context with trainable long-context variants at 16K and 64K using PI/YaRN; NTK can push the 16K models to roughly 24K–32K at inference.
• FlashAttention-2 used for training all models to keep memory from exploding as contexts grow.
• RLHF-tuned Chinese-Alpaca-2-RLHF models (1.3B, 7B) targeting better alignment with human preferences on correct values.
• Supports downstream tooling including transformers, llama.cpp, vLLM, LangChain, and privateGPT.
• Includes quantization recipes for offline CPU/GPU deployment.

Caveats

• The project has been superseded by Chinese-LLaMA-Alpaca-3 (based on Llama-3), which the authors explicitly recommend for new work.
• The 64K context models are only available at 7B; 13B stops at 16K.
• RLHF details are sparse in the README; the exact dataset and training recipe for alignment are not described beyond the high-level claim.

Verdict

Worth a look if you need a fully open, documented Chinese Llama-2 variant with long-context training already baked in. Skip it if you are starting fresh—the authors themselves point to their Llama-3-based third generation as the better foundation.