Your face, their moves, no 3D rig required

What it does

Feed it a source image and a driving video, and it spits out a new video where the subject in your photo mimics the motion from the clip. It learns unsupervised keypoints directly from raw video frames and transfers relative movement between them, preserving the source appearance. The authors trained separate models for different domains—faces, fashion, and human actions—so a one-size-fits-all checkpoint is not the point.

The interesting bit

The system skips explicit 3D modeling and annotated landmarks entirely. It estimates how keypoints move in the driving video, then applies those same displacements to keypoints detected in the static source image. This relative coordinate trick usually outperforms absolute positioning, but it comes with a catch: the source subject and the driving video need to start in roughly the same pose, or the illusion breaks.

Key highlights

• Handles faces (VoxCeleb), fashion, and human motion datasets like Taichi and MGIF via domain-specific trained models
• Relative motion transfer generally beats absolute mode, which the authors warn tends to transfer irrelevant shape details
• Pre-trained checkpoints and a Hugging Face Spaces demo lower the barrier to a quick trial
• Can be extended to face-swap tasks when paired with supervised segmentation masks
• Default training configuration targets multi-GPU setups (batch sizes tuned for 2–4 Titan-X cards)

Caveats

• Source images and driving videos must be cropped and aligned beforehand; raw footage will not work out of the box
• The relative animation mode requires the source subject and driving video to share the same initial pose
• The authors now point to a newer framework for articulated objects, suggesting this model is showing its age on complex skeletons

Verdict

A solid starting point if you want motion retargeting for faces or fashion without building a 3D pipeline. Look elsewhere if you need production-ready articulated body animation or want to drop in unprocessed video clips.