Turn a 2D scribble into an orbitable 3D neural field

What it does

Pix2pix3D is a conditional generative model that takes a 2D label map—think segmentation masks or edge drawings—and synthesizes a neural radiance field representing a 3D object. It renders both a photorealistic RGB image and a pixel-aligned label map from arbitrary camera viewpoints. The project ships with pretrained models for cats, faces, and cars, plus scripts to generate stills, videos, and semantic meshes.

The interesting bit

Instead of generating a flat image, the model assigns a semantic label to every 3D point alongside color and density, so the same label map guides synthesis consistently across viewpoints. This lets users edit the label map from one angle and regenerate the object from another without losing coherence.

Key highlights

• Built on top of EG3D and StyleNeRF, extending them with 3D-aware conditional control
• Supports segmentation-to-image (seg2cat, seg2face) and edge-to-image (edge2car) synthesis
• Can extract colored semantic meshes in .ply format using rendered 3D labels
• CVPR 2023 implementation from CMU; includes pretrained checkpoints

Caveats

• Mesh extraction requires installing extra packages (pyrender, trimesh, mcubes) beyond the base environment
• The README advertises an interactive 3D editing demo but never explains how to launch it
• Training partially initializes from EG3D checkpoints and the example script assumes 2 GPUs

Verdict

Worth a look if you research 3D-aware generation or need to bootstrap labeled NeRFs from 2D annotations. Skip it if you want a lightweight, single-GPU training pipeline or a fully documented interactive UI.