Teaching LLaMA Chinese law, one LoRA at a time

What it does

LaWGPT is a family of 7B-parameter Chinese legal language models created by continuing pre-training on legal corpora and then instruction-tuning on legal Q&A and national judicial-exam datasets. Because the underlying Chinese-LLaMA and LLaMA weights are not redistributable, the project releases only LoRA adapters; users must merge them with the original base weights locally. The resulting model answers questions about drunk-driving sentences, gambling definitions, and overtime pay in Chinese.

The interesting bit

The authors use ChatGPT to clean and polish their instruction-following training set—essentially hiring a generalist model as a teaching assistant for a specialist student. They also expanded the tokenizer with a domain-specific legal vocabulary before pre-training, which is the kind of unglamorous preprocessing that usually determines whether a domain model actually understands specialized terms or just dresses up generic answers in legal jargon.

Key highlights

• Two-stage pipeline: continued pre-training on Chinese legal texts and statutes, followed by supervised fine-tuning on up to 350,000 legal Q&A pairs and judicial-exam questions.
• Ships as LoRA weights only; requires separately licensed Chinese-LLaMA/Alpaca base weights for merging.
• Includes a Gradio web UI and batch-inference scripts for testing.
• Trained on 8× Tesla V100-SXM2-32GB GPUs; pre-training takes roughly 24 hours per epoch, fine-tuning about 12.
• Explicitly restricted to academic research; the authors prohibit commercial use and warn against using outputs in real legal proceedings.

Caveats

• The authors acknowledge weak factual recall and limited capacity, so the model may hallucinate statutes or misstate sentencing guidelines.
• Human-intent alignment is preliminary, meaning it can still produce unpredictable or harmful content.
• Because only LoRA weights are provided, you need access to the separately licensed base models, which adds friction.

Verdict

A useful reference if you are researching Chinese legal NLP or domain-specific fine-tuning recipes. Avoid it if you need a production-ready legal advisor or a turnkey model you can download and run immediately.