Differentiable 3D rendering: backprop through pixels to meshes
A PyTorch port of a CVPR 2018 renderer that lets you optimize vertices, textures, and camera pose via gradient descent.

What it does This is a PyTorch reimplementation of the Neural 3D Mesh Renderer, originally written in Chainer by the paper authors. It rasterizes 3D meshes into 2D images with a twist: the rendering pipeline is differentiable, so you can backpropagate image-space loss all the way back to mesh vertices, textures, or camera parameters.
The interesting bit Standard renderers drop gradients at the visibility boundary, which breaks end-to-end training. This one approximates those gradients, letting you do things like morph a teapot silhouette into a rectangle or recover camera pose purely through pixel comparison and gradient descent.
Key highlights
- Ships with four working examples: multi-view rendering, vertex optimization, texture matching, and camera parameter recovery
- Adds small API extensions over the original: general 3×4 camera matrix support, lens distortion coefficients
- Passes all test cases from the original Chainer implementation
- Installable via
pip install neural_renderer_pytorch
Caveats
- Locked to PyTorch 0.4.0; newer versions may hit
AT_ASSERTcompilation errors (fix exists on a branch, not yet merged to master) - “Detailed documentation will be added in the near future” — for now, the examples are your docs
- API may change; this is a research port, not a stable library
Verdict Grab it if you’re doing differentiable rendering research or need to backprop through a rasterizer. Skip if you want production-grade 3D graphics or modern PyTorch support out of the box.