NeRFs That Talk, but Only Where It Matters

What it does ER-NeRF learns a photorealistic, audio-driven talking portrait from a short training video of a speaker. It uses Neural Radiance Fields to model the subject, then renders new footage synchronized to arbitrary audio input, with separate training stages for the head, lip detail, and torso.

The interesting bit Instead of burning compute uniformly across the entire frame, the method is region-aware: it concentrates modeling power on the face and mouth where motion happens, leaving static regions alone. The codebase is essentially a heavy fork of RAD-NeRF, borrowing data pipelines from AD-NeRF and GeneFace, so the novelty lies in the architecture rather than the scaffolding.

Key highlights

• Official ICCV 2023 implementation; pretrained Obama checkpoint available with reported metrics (head PSNR ~35.6, LPIPS ~0.018)
• Flexible audio encoding: supports DeepSpeech (paper default), Wav2Vec, and HuBERT for broader language coverage
• Three-part training pipeline: base head model, lip finetuning with patch-based LPIPS, and optional full torso synthesis
• Rigid data requirements: input must be 25 FPS, ~512×512, 1–5 minutes long, with the speaker visible in every single frame

Caveats

• Dependency stack is frozen in time: PyTorch 1.12, CUDA 11.3, TensorFlow 2.8.0, and custom CUDA extensions that compile on first run
• Preprocessing is a slog: several hours of processing, external 3DMM downloads (Basel Face Model), OpenFace AU45 extraction for blinks, and manual video cropping
• Authors have already superseded this with newer work (TalkingGaussian, InsTaG), and a TODO admits full AU-driven expression control is still missing

Verdict A solid reference for researchers benchmarking ICCV 2023 NeRF-based lip-sync methods. Casual users and anyone without an Ubuntu 18.04 time machine should consider the authors’ later Gaussian splatting approaches instead.