GLM unifies NLP by treating every task as blank-filling

What it does

GLM is a family of pretrained language models—from 110M to 10B parameters, in both English and Chinese—that share one autoregressive blank-filling objective. Instead of splitting understanding and generation across separate architectures, it masks contiguous text spans and trains the model to fill them in left-to-right. The resulting checkpoints can be fine-tuned for SuperGLUE classification, summarization, or open-ended text generation.

The interesting bit

GLM uses three special mask tokens—[MASK], [sMASK], and [gMASK]—to signal whether it should fill a short blank, a whole sentence, or generate left-to-right. By varying the number and length of masked spans, the same pretrained backbone handles NLU, conditional generation, and unconditional generation without separate encoder-decoder or autoregressive stacks.

Key highlights

• On SuperGLUE, CNN/Daily Mail, XSum, and LAMBADA, the 10B model goes toe-to-toe with DeBERTa-XXLarge, T5-11B, PEGASUS, and GPT-2.
• Weights are hosted on Tsinghua Cloud and OneDrive, and the models are loadable via Hugging Face transformers>=4.23.1 with trust_remote_code=True.
• Supports multi-GPU model parallelism via checkpoint splitting, though it requires manual config edits.
• Docker images ship for CUDA 10.2 and 11.2, but manual installation pins PyTorch to 1.7.0 and requires NVIDIA Apex.

Caveats

• The Hugging Face integration requires trust_remote_code=True, meaning you are running unaudited custom model code.
• The codebase targets an older stack (PyTorch 1.7.0, CUDA 10.2/11.2), so expect friction on modern hardware.
• Enabling model parallelism involves hand-editing shell scripts and config files rather than flipping a single flag.

Verdict

Grab this if you are reproducing the ACL 2022 paper or building on the GLM lineage such as ChatGLM. If you want a plug-and-play foundation model for a new product, newer and more polished alternatives exist.