Giving FLAME a Skin: Building a Texture Space from Casual Photos

What it does

The repository implements an analysis-by-synthesis pipeline that fits a textured version of the FLAME 3D head model to in-the-wild facial images. It optimizes shape, pose, expression, lighting, and appearance parameters to match a photo, and it documents how to build a PCA-based texture space for FLAME from 1,500 FFHQ images. FLAME itself ships without an appearance model, so this work effectively fills that gap by supplying the missing appearance data.

The interesting bit

Building a texture space from unconstrained internet photos is a chicken-and-egg problem: you need a texture model to fit the images, but you need fitted images to build the model. The authors bootstrap an initial texture basis from the Basel Face Model, fit it to FFHQ, then use an adapted GMCNN inpainting network to hallucinate the missing parts of each texture map before running PCA. It is a pragmatic pipeline of optimization, segmentation masks, and neural inpainting rather than a single end-to-end network.

Key highlights

• Optimizes FLAME geometry, expression, spherical-harmonics lighting, and texture against a single photo using skin-region photometric loss; occlusions are masked out via a segmentation network.
• Completes partial texture maps with an inpainting network trained to fill random strokes in visible face regions, then derives a PCA texture space from 1,500 finished maps.
• Relies on predicted 2D landmarks (FAN) to anchor the model fitting and regularizes shape, pose, and appearance to keep reconstructions plausible.
• Built on PyTorch and PyTorch3D (specifically versions 1.5 and 0.2) and reuses components from FLAME_PyTorch and DECA.
• Both the code and the resulting texture model are strictly for non-commercial research use.

Caveats

• The README warns that “demos will be released soon,” which suggests some promised examples may still be pending even though a single-image fitting script is already present.
• The core FLAME model and texture space require a manual licensed download from the FLAME website; the code does not run standalone without them.
• The implementation targets older PyTorch and PyTorch3D releases, so modern environments may need dependency tweaks.

Verdict

A solid resource for computer-vision researchers who need a controllable, optimizable face appearance model built from real-world diversity. Skip it if you are looking for a commercial, ready-to-deploy product or a fully end-to-end deep-learning shortcut.