Llama 3 learns Chinese without expanding its vocabulary

What it does

Chinese-LLaMA-Alpaca-3 is the third installment in a series adapting Meta’s Llama 3 for Chinese. It releases an 8B base model incrementally pre-trained on roughly 120 GB of unlabeled Chinese text, plus three instruction-tuned variants (v1, v2, v3) refined on millions of instruction samples. The project also ships training scripts, curated datasets like alpaca_zh_51k and ruozhiba_gpt4, and quantized GGUF weights for local CPU or GPU deployment via llama.cpp, vLLM, Ollama, and transformers.

The interesting bit

The team deliberately kept Meta’s original 128K BPE vocabulary. They found Llama 3’s native encoding efficiency is about 95% of their own previously expanded Chinese-LLaMA-2 tokenizer, so expanding the dictionary wasn’t worth the compatibility headache. The latest Instruct-v3 model is a fusion of earlier versions plus a light fine-tuning pass on about 5,000 extra instructions, and it currently leads the pack with an 83.6% arena win rate and the highest C-Eval scores of the series.

Key highlights

• Releases full weights, LoRA adapters, and GGUF quantized versions for consumer hardware
• Context window doubled to 8K from the previous generation’s 4K, with support for PI/NTK/YaRN extensions
• Uses Llama 3’s native grouped-query attention and its new instruction template (incompatible with Llama 2 chat formats)
• Open-sources pre-training and fine-tuning scripts alongside curated instruction datasets
• Instruct-v3 outperforms v1 and v2 on Chinese (C-Eval) and English (Open LLM Leaderboard) benchmarks

Caveats

• Instruct-v3’s long-text score on LongBench is 40.5, trailing v2’s 46.4, so the “best” version depends on your use case
• Only 8B parameter models are released; no larger variants are mentioned in the README

Verdict

Worth a look if you need a drop-in Chinese-speaking Llama 3 that runs on consumer hardware and plays nicely with existing tooling. Skip it if you require a custom tokenizer or long-context performance above all else.