Google shrinks BERT by sharing parameters across every layer

What it does ALBERT is a Transformer-based language representation model that cuts parameter count by sharing weights across all layers and factorizing embedding matrices. The README says this overcomes previous memory limitations and lets you train larger configurations than standard BERT without the usual degradation. It ships with pre-trained checkpoints (Base through Xxlarge), including Chinese variants, and fine-tuning scripts for GLUE, SQuAD, and RACE.

The interesting bit The v2 release is a case study in training dogma: the authors removed dropout, added data, and trained longer. Base through Xlarge models improved markedly, but the Xxlarge model barely budged—suggesting that at a certain scale, more steps just burn electricity. The authors openly note that downstream tasks are hypersensitive to fine-tuning hyperparameters, so the leaderboard gains come with a tuning tax.

Key highlights

• Parameter sharing and embedding factorization let ALBERT scale to Xxlarge while keeping the memory footprint lower than naive BERT growth.
• On GLUE, the 1.5M-step ALBERT-xxlarge edges out RoBERTa-large and XLNet-large in every listed task.
• Supports TF-Hub modules and raw checkpoints, plus a Colab tutorial for GLUE fine-tuning.
• Includes dedicated fine-tuning scripts for SQuAD v1/v2, RACE, and GLUE tasks.
• Chinese pre-trained models are available courtesy of the CLUE team.

Caveats

• The repository targets TensorFlow 1.x; v2 models were specifically patched to remove a native Einsum op for TF 1.15 compatibility, which gives the codebase a distinctly 2019 aroma.
• The authors caution that fine-tuning hyperparameters can make or break results, and v2 models diverge with v1 RACE settings.

Verdict Researchers and practitioners still working in the TensorFlow 1.x ecosystem who need a proven, pre-trained encoder with a smaller parameter budget should look here. If you are hunting for modern PyTorch-native implementations or TF 2/Keras pipelines, this is a reference implementation, not a framework.