From zero to GLM4: a Chinese cookbook for hand-rolling transformers

What it does This is a Chinese-language educational repo that walks you through building large language models from literal scratch in PyTorch. It splits into two tracks: a foundational course where you implement a GPT-like model through text processing, attention, pretraining, and generation; and an architecture deep-dive where you dissect ChatGLM3/4, Llama3, MiniCPM, and RWKV V2 through V6. Everything is delivered as runnable Jupyter notebooks aimed at making the internals legible rather than production-ready.

The interesting bit Instead of stopping at a toy GPT, the project treats mainstream open-weight models as homework assignments. It builds its fundamentals track on Sebastian Raschka’s LLMs-from-scratch, then extends that approach into a broader survey of Chinese and Western model families.

Key highlights

• Foundational notebooks cover text preprocessing, multi-head attention, a full GPT implementation, and pretraining loops.
• Architecture-specific notebooks for ChatGLM3/4, Llama3, MiniCPM, and RWKV V2–V6, including weight-loading and configuration examples.
• Two difficulty modes: quick-start code in Codes/ and expanded explanatory text in Translated_Book/.
• Explicitly educational scope: targets learners with basic PyTorch knowledge, not researchers hunting for training frameworks.

Caveats

• Chapters 6 through 8 (fine-tuning for classification, RLHF, and production usage) are listed as “即将发布” — not yet available.

Verdict Worth bookmarking if you are a Mandarin-speaking developer who learns by typing, or if you need to onboard a team onto transformer internals. Skip it if you are looking for a drop-in training framework or already hand-roll MoE architectures for fun.