Attention Is All You Need, in Code You Can Read

What it does This is a ground-up TensorFlow implementation of the 2017 “Attention Is All You Need” architecture. It trains a German-to-English translation model on the IWSLT 2016 dataset and reports BLEU scores of 28.06 on the dev set and 23.88 on test. The author originally wrote it when no official code existed, then revised it to fix early bugs in masking and positional encoding while keeping the codebase compact and heavily commented.

The interesting bit Rather than competing with Google’s tensor2tensor on massive datasets, the author keeps the scope deliberately small—modular but not over-engineered—so you can verify that your Transformer actually works without drowning in infrastructure. It functions like a controlled lab notebook for a paper that changed NLP.

Key highlights

• Clean, modular TF 1.12 code with extensive inline comments
• Fixes known early-implementation bugs (masking, positional encoding)
• Adds BPE tokenization and shared weight matrices that were missing in the original release
• Pretrained checkpoints and IWSLT 2016 de-en data pipeline included
• Achieves 28.06 BLEU on dev, 23.88 on test

Caveats

• Locked to TensorFlow 1.12; the README only tentatively mentions a future TF 2.0 update
• Author explicitly recommends the official tensor2tensor implementation for large-scale work like WMT
• Beam decoding is noted as missing

Verdict Grab this if you want to read a Transformer implementation line-by-line without parsing a research framework. Skip it if you need production-scale training or a modern TF 2.x / PyTorch stack.