Inverse rendering that outputs actual triangle meshes

What it does Nvdiffrec takes multi-view photographs and jointly reconstructs a triangular 3D mesh, spatially-varying materials, and environment lighting. It frames the task as a large-scale inverse rendering problem, optimizing topology and surface appearance simultaneously to match the input observations. The output is an explicit triangular model rather than a neural density field.

The interesting bit Instead of stopping at a signed distance field or NeRF-like density, the pipeline extracts actual triangle topology through differentiable marching tetrahedrons adapted from NVIDIA Kaolin. A later update added FlexiCubes support, giving the mesh extraction step an alternative isosurfacing backend. The project also maintains a slang branch that replaces hand-rolled CUDA extension gradients with SlangPy autodiff, trading boilerplate for the same runtime.

Key highlights

• Joint optimization of mesh topology, materials, and lighting from image observations
• Differentiable marching tetrahedrons for end-to-end topology learning
• Optional FlexiCubes backend for isosurfacing
• slang branch simplifies custom renderutils by using SlangPy autodiff instead of manual CUDA forward/backward passes
• Includes reproduction configs for NeRF synthetic and NeRD datasets

Caveats

• Requires a high-end NVIDIA GPU with substantial VRAM; mid-range cards need batch-size reduction
• Multi-GPU training is experimental and Linux-only; all published results used a single GPU
• Live training visualization is only supported on Windows

Verdict Worth a look if you need explicit mesh extraction from photos and have the GPU hardware to match. Skip it if you want a lightweight photogrammetry tool or a permissive open-source license.