An Open Legal MoE That Claims to Beat GPT-4 on Chinese Bar Exams

What it does

ChatLaw is a family of open models tuned for Chinese legal work. The flagship ChatLaw2-MoE runs a 4×7B Mixture-of-Experts architecture on top of InternLM, while older demos include 13B and 33B variants built on Ziya-LLaMA and Anima. A companion ChatLaw-Text2Vec model, trained on 93,000 court decisions, matches user queries to relevant statutes rather than generating answers from scratch. The system is designed as a multi-agent pipeline—information gathering, legal research, and final report generation—mimicking the workflow of an actual law firm.

The interesting bit

The project treats hallucination as a process problem, not just a model problem. It pairs a knowledge-graph-enhanced MoE with Standardized Operating Procedures inspired by real law-firm workflows, routing tasks through multiple collaborative agents. The authors claim this setup outperforms GPT-4 on the Lawbench benchmark and the Unified Qualification Exam for Legal Professionals by 7.73% and 11 points respectively.

Key highlights

• Multi-agent pipeline: Breaks a consultation into discrete steps (intake, research, advice, report) rather than firing a single prompt at one model.
• MoE architecture: The current ChatLaw2-MoE uses a 4×7B sparse expert design, which is less common in legal-domain open models.
• Statute retrieval: ChatLaw-Text2Vec grounds responses by linking queries to specific contract-law sections via similarity search over 93k court cases.
• Benchmark claims: The README cites top scores on legal cognitive tasks, five years of bar-exam data, and a TruthfulQA evaluation—though specific TruthfulQA numbers are not shown in the text.

Caveats

• The 13B and 33B versions are explicitly labeled as demos; the 33B variant occasionally drifts into English responses due to limited Chinese training data in its Anima base.
• The repo is primarily a model card and paper landing page; there is no visible training code, inference server, or dataset download in the README.

Verdict

Worth a look if you are building Chinese legal-tech tools or researching domain-specific MoE architectures. Skip it if you need a turnkey API or want to inspect the training pipeline—the README points you to a HuggingFace model page and little else.