wdndev/tiny-llm-zh
A from-scratch implementation of a small-parameter Chinese large language model covering tokenizer training, pretraining, instruction tuning, RLHF/DPO alignment, and inference serving.
Velocity · 7d
+1.3
★ / day
Trend
→steady
star history
This project implements a small Chinese LLM from scratch with a complete training pipeline. It trains a custom tokenizer on 10GB of Chinese corpus (20K vocab merged with Llama2), performs pretraining on 42B tokens, instruction fine-tuning on 400K examples, and alignment via RLHF/DPO on 170K samples. The repository supports multiple model sizes (16M–440M parameters), MoE architecture, distributed training via Transformers and DeepSpeed, and inference via vLLM and llama.cpp.