The 2018 BERT reimplementation that prioritized readability over rigor

What it does This is a PyTorch reimplementation of Google AI’s 2018 BERT paper, covering the two original pre-training tasks: masked language modeling and next-sentence prediction. It packages the model and training logic into an installable library with command-line tools for vocabulary building and model training. The author’s stated goal was a simple, heavily annotated codebase for fast comprehension.

The interesting bit Written in the didactic spirit of Harvard’s Annotated Transformer, this was among the first community attempts to translate BERT into PyTorch while the paper was still fresh. It favors legibility over engineering polish, making the transformer mechanics transparent rather than opaque.

Key highlights

• Implements both core BERT pre-training objectives: masked LM (with the 80/10/10 masking split) and next-sentence prediction.
• Exposes CLI tools (bert-vocab, bert) via a PyPI package.
• Explicitly modeled after The Annotated Transformer for readability.
• Apache 2.0 licensed.

Caveats

• The README plainly states the project is a work in progress and the code is “not verified yet.”
• Tokenization is not included; the pipeline expects tab-separated sentence pairs or already-tokenized input.
• It is positioned as a learning resource, not a production framework.

Verdict Pick this up if you want to read a bare-bones BERT implementation from the early post-paper era. Avoid it if you need a finished, validated training pipeline—the author explicitly warns it is neither.