A warts-and-all PyTorch port of Google’s DreamFusion

What it does

It trains a NeRF representation guided by a 2D diffusion model to turn text prompts—or a single reference image—into a 3D asset you can export as a mesh or spin around in a GUI. The core trick is the same as DreamFusion: render the NeRF from a random angle, check if the diffusion model thinks the image matches the prompt, and backpropagate the disagreement into the 3D volume. You can also refine the resulting geometry with DMTet for higher-resolution surfaces.

The interesting bit

Instead of Google’s proprietary Imagen, it swaps in Stable Diffusion (and optionally DeepFloyd-IF), which forces the loss to travel backward through the VAE encoder since SD works in latent space. That extra hop costs training time, but it means the whole stack runs on standard consumer-ish hardware—about 16 GB of VRAM with the Instant-NGP backbone, or a CUDA-free Taichi backend if you prefer not to compile extensions.

Key highlights

• Supports both text-to-3D and image-to-3D via Zero-1-to-3, with optional monocular depth and normal preprocessing.
• Uses an Instant-NGP grid encoder for ~10 FPS rendering at 800×800, or a vanilla pure-PyTorch NeRF backbone if you want to avoid CUDA builds entirely.
• Includes Perp-Neg guidance to reduce the classic “multi-head” artifact where text-to-3D models spawn extra faces.
• Exports textured meshes (OBJ, MTL, and PNG) and interactive GUI inspection after training.
• Experimental DeepFloyd-IF backend and DMTet finetuning for mesh-aware refinement.

Caveats

• The author explicitly warns that generation quality does not match the original paper and many prompts fail badly.
• The repo itself is deprecated in spirit: the README advertises threestudio as the successor with better implementation and recent improvements.
• Image-to-3D is tightly coupled to Zero-1-to-3’s capabilities and requires carefully preprocessed, front-facing, square images under 1024 pixels.

Verdict

Tinkerers and researchers who want to see how DreamFusion’s sausage is made—or need a hackable baseline—will find a transparent, if flawed, reference implementation. Anyone seeking production-ready 3D generation should follow the author’s own advice and look at threestudio instead.