A field guide to GAN training folklore from NIPS 2016

What it does

This repository is a curated cheat sheet of seventeen practical heuristics for training Generative Adversarial Networks without them collapsing or suffering vanishing gradients. Born from a NIPS 2016 workshop and authored by researchers including Martin Arjovsky and Soumith Chintala, it reads like field notes from the early days of generative modeling committed to Markdown. There is no codebase—just hard-won rules about normalization, loss functions, optimizers, and architecture choices.

The interesting bit

The document’s candor is its real value: the authors openly mark some tricks as uncertain ("[notsure]"), admit that balancing generator and discriminator schedules is a trap “we’ve all tried,” and warn that a steadily decreasing generator loss usually means it is “fooling D with garbage.” That kind of unvarnished field reporting is rare in formal literature.

Key highlights

• Normalize inputs to [-1, 1] and cap the generator with Tanh, but sample latent vectors from a Gaussian sphere rather than a uniform cube.
• Keep real and fake images in separate mini-batches when using BatchNorm, or fall back to instance normalization.
• Flip labels when training the generator and use soft labels (e.g., real = 0.7–1.2, fake = 0.0–0.3) to keep the discriminator from getting too confident.
• Borrow stability tricks from reinforcement learning, such as experience replay buffers and occasionally swapping past checkpoints of G and D.
• Use dropout at both train and test time in the generator to inject noise, and prefer LeakyReLU over ReLU to avoid sparse gradients.

Caveats

• The authors explicitly note the list is “no longer maintained” and are “not sure how relevant it is in 2020,” so some advice may be stale for modern generative workflows.
• Two entries are tagged “[notsure]” by the authors themselves—training the discriminator more often, and batch discrimination—because the results were mixed or hard to tune.
• Several tips are presented as empirical folklore rather than theoretically justified guarantees.

Verdict

Worth a skim if you are still training old-school GANs or studying deep generative history; skip it if you need maintained code or guarantees that these tricks still beat modern baselines.