Sparse views, dense bodies: a diffusion model for 4D humans

What it does Diffuman4D takes sparse multi-view recordings of a human performance—as few as four cameras—and generates the missing viewpoints across time, producing a dense set of images suitable for free-viewpoint rendering. The output can be fed into 3D Gaussian Splatting or 4D Gaussian Splatting pipelines to reconstruct explorable dynamic human models.

The interesting bit Instead of treating each frame or camera independently, the model jointly reasons over space and time, which is where most novel-view synthesis methods fall apart on dynamic bodies. The authors back this with a meticulously reprocessed version of the DNA-Rendering dataset, releasing over 1,000 sequences with recalibrated cameras, color correction, foreground masks, and skeletons.

Key highlights

• Generates dense novel views from sparse input (e.g., 4 cameras to 44) across full video sequences.
• Explicitly targets 4D consistency, keeping synthesized geometry and appearance stable through both camera motion and time.
• Ships with a fully processed, re-annotated DNA-Rendering dataset containing over 10 million images, masks, 2D/3D poses, and camera parameters.
• Provides an interactive web demo and pretrained checkpoints for immediate inference.
• Includes preprocessing scripts for custom multi-view data, though keypoint extraction requires a separate Sapiens environment.

Caveats

• The alternative 4D Gaussian Splatting reconstruction scripts are not yet released; the README says the team will “attempt to provide” them since the preferred LongVolcap pipeline remains closed-source.
• Custom data preprocessing demands a separate conda environment for Sapiens-lite pose prediction due to PyTorch version constraints.
• Inference assumes Hugging Face access for model downloads, with manual fallback only mentioned as a workaround for network issues.

Verdict Worth exploring if you are building human-centric 4D reconstruction or view-synthesis pipelines and already have synchronized multi-camera data. Skip it if you are looking for a single-video avatar generator or a fully turnkey 4DGS training pipeline.