Bilingual LLMs trained from scratch, Llama-shaped but not Llama-bred

What it does Yi is a family of large language models developed by 01.AI and trained from scratch on a 3 trillion token multilingual corpus. The release includes base and chat variants at 6B, 9B, and 34B parameters, with standard context windows up to 4K and extended 200K versions. Quantized 4-bit and 8-bit chat models are available, along with Yi-VL vision-language variants.

The interesting bit The models adopt Llama’s architecture for toolchain compatibility but are explicit about not using Llama’s weights or data. 01.AI insists the training data, pipelines, and infrastructure were built independently, which is the unglamorous foundation behind its claims of strong bilingual performance.

Key highlights

• Bilingual targeting: optimized for both English and Chinese evaluation suites
• Dated but specific benchmark claims: Yi-34B-Chat reportedly placed second to GPT-4 Turbo on AlpacaEval (as of Jan 2024), while Yi-34B led open-source models on Hugging Face and C-Eval (as of Nov 2023)
• Long-context results: Yi-34B-200K claims a 99.8% score on the Needle-in-a-Haystack test after continued pre-training on 5B long-context tokens
• Drop-in compatibility: works with llama.cpp, Hugging Face, and standard quantization tools despite the independent weights
• Ecosystem breadth: models are mirrored on Hugging Face, ModelScope, and wisemodel, with chat, base, vision, and quantized flavors

Caveats

• Leaderboard positions are explicitly time-boxed to late 2023 and early 2024; the README offers no comparison against releases from mid-2024 onward
• The repository centers on model distribution and usage guides rather than training frameworks or novel inference code

Verdict Evaluate Yi if you need a Llama-compatible bilingual model with independently trained weights. Look elsewhere if you want the underlying training stack or fresh benchmark data against the newest open-source releases.