NVIDIA's Kaolin bakes differentiable rendering into PyTorch

What it does Kaolin is a PyTorch-native library that collects GPU-optimized building blocks for 3D deep learning: modular differentiable rendering, fast conversions between meshes, voxels, point clouds, and Gaussian splats, plus a differentiable camera and lighting API. It also packs batched mesh containers, a physics module with collision support, and an interactive Jupyter visualizer—essentially the standard-library plumbing that 3D ML papers usually re-implement from scratch.

The interesting bit The library is actively absorbing new 3D generative trends: recent releases added 3D Gaussian-to-voxel conversion for densification and collision-aware physics simulation. Even FlexiCubes, NVIDIA’s gradient-based mesh optimization method, recently migrated from a restrictive non-commercial license to Apache 2.0.

Key highlights

• GPU-optimized differentiable rendering, camera, and lighting APIs (spherical harmonics and spherical gaussians)
• Structured Point Clouds, a quadtree acceleration structure for spatial queries
• Built-in physics simulation with collision support as of v0.18.0
• 3D Gaussian splatting utilities including conversion to voxel grids
• Interactive 3D visualizer for Jupyter notebooks
• Default imports are Apache 2.0; legacy components remain under a non-commercial NSCL license in kaolin/non_commercial

Caveats

• Some legacy components still carry a non-commercial NSCL license, though the main FlexiCubes implementation is now Apache 2.0
• The README reads like a feature inventory and external project index; it offers no internal benchmarks or comparative performance data against other 3D frameworks

Verdict Researchers building differentiable pipelines over meshes, splats, or implicit fields should treat Kaolin as essential infrastructure. Purely 2D practitioners, or anyone who needs a fully permissive codebase without checking import paths, should look elsewhere.