Face swaps with adversarial loss and a steady gaze

What it does

faceswap-GAN is a collection of Jupyter Notebooks that train a denoising autoencoder to swap one person’s face onto another in images and video. It builds directly on the architecture popularized by the original reddit deepfakes project, but layers in adversarial loss, VGGFace perceptual loss, and a learned attention mask to sharpen output and stabilize artifacts. The pipeline covers the full workflow: preprocessing video with MTCNN face detection and alignment, training the model at resolutions up to 256×256, and converting frames back into a final video with Kalman-filtered bounding boxes to reduce jitter.

The interesting bit

The project’s real obsession is eyes. It uses eyes-aware training—higher reconstruction and edge loss around the eye region—and VGGFace perceptual loss to nudge eyeball direction toward realism, which is exactly where most face swaps look uncanny. It also adopted a self-attention mechanism from SAGAN before official code was available, though the author candidly admits that implementation might be off.

Key highlights

• Adds adversarial and VGGFace perceptual loss to the classic deepfakes autoencoder architecture.
• Predicts an attention mask to handle occlusion and smooth skin-tone transitions.
• Supports 64×64, 128×128, and 256×256 output resolutions in version 2.2.
• Video pipeline uses MTCNN for face alignment and a Kalman filter to stabilize bounding boxes across frames.
• Eyes-aware training with boosted reconstruction and edge loss targets realistic gaze and blink.

Caveats

• The self-attention mechanism was implemented before the official SAGAN code dropped, and the author notes it “could be wrong.”
• As of late 2019, the provided Colab notebooks throw import errors because the pinned dependencies—Keras 2.1.5 and TensorFlow 1.6.0—have aged out of the default environment.
• The workflow is notebook-centric and requires manual shuffling of aligned face images between folders.

Verdict

Worth a look if you are studying the evolution of deepfake architectures or need a hackable, end-to-end notebook pipeline for research. Skip it if you want a maintained library or a one-click install; this is a 2018-era research artifact with brittle dependencies.