Open bilingual LLMs benchmarked on law, medicine, and high school

What it does

Baichuan 2 is a family of large language models—7B and 13B parameter base and chat variants—trained by Baichuan Intelligent Technology on 2.6 trillion tokens. The repository distributes model weights, 4-bit quantized chat versions, intermediate checkpoints, and exhaustive self-reported benchmarks against LLaMA 2, ChatGLM2, GPT-3.5 Turbo, and GPT-4.

The interesting bit

The evaluation suite is heavily tuned to Chinese contexts: the models sit for China’s national judicial exam (JEC-QA), the gaokao college-entrance test, and the CMMLU Chinese-knowledge benchmark alongside English standards like MMLU. The README claims these are the best results for their parameter class. Less common is the licensing: commercial use is free after an email application, which is more permissive than many similarly capable model releases.

Key highlights

• 7B and 13B base and chat weights, plus 4-bit quantized chat variants
• Trained on 2.6 trillion tokens; intermediate checkpoints available
• Benchmarked across general knowledge, law, medicine, math, code, and translation
• Claims top marks among same-size peers on C-Eval, CMMLU, and Gaokao, while trailing GPT-4 and GPT-3.5 Turbo
• Free academic use; free commercial licensing via email application

Caveats

• The README is Chinese-first, so English-speaking developers will need to rely on the linked English translation or browser tools
• The provided documentation is heavy on benchmark tables and light on architecture or training methodology, which is offloaded to an external technical report

Verdict

Grab it if you need a Chinese-optimized, commercially usable LLM in the 7B–13B range. Look elsewhere if you want deep training transparency or an English-native developer experience.