Backpropagating diffusion scores into 3D voxel space

What it does SJC generates 3D assets from text prompts by treating a pretrained 2D diffusion model as a gradient oracle. It renders a voxel radiance field from multiple camera angles, feeds those 2D views into a diffusion model like Stable Diffusion, and backpropagates the diffusion score through the renderer’s Jacobian to update the 3D volume. The result is a 3D-aware optimization that assembles consistent geometry without any native 3D training data. The repo also bundles a Karras sampler and a lightweight voxel NeRF implementation.

The interesting bit The core trick is applying the chain rule to the diffusion model’s learned score function—essentially treating the renderer as a differentiable layer between 3D parameters and 2D pixels. The authors identify a distribution mismatch that arises when naively aggregating 2D scores into 3D, and propose a specific estimation mechanism to handle it; the README notes you can toggle between two variance-reduction equations (var_red) depending on the prompt.

Key highlights

• Generates 3D objects from text prompts using Stable Diffusion or other off-the-shelf 2D diffusion checkpoints
• Aggregates multi-view 2D diffusion scores into a unified 3D score via Jacobian backpropagation through a voxel radiance field
• Includes a subpixel rendering script to clean up jittery final visualizations (about 5 minutes / 11 GB on an A5000)
• Also implements a Karras sampler and a simplified voxel NeRF in the same codebase
• Integrated into the larger threestudio framework

Caveats

• All random seeds are currently hardcoded to 0, and a script to reproduce the 2D experiments is still missing (per the TODO list)
• DreamBooth integration is explicitly noted as “not ready,” and fine-tuned models may need lower guidance scales to avoid multi-face artifacts
• Main paper figures were not rendered with the newer subpixel pipeline, so published visuals differ slightly from the current recommended output

Verdict Researchers and practitioners interested in score distillation or text-to-3D generation should look here for an early, well-documented reference implementation. If you need a polished, end-user text-to-3D tool, this is research code with rough edges and hardcoded seeds.