Trading neural radiance fields for fuzzy 3D Gaussians

What it does

This is the authors’ official implementation of an ACM TOG 2023 paper that reconstructs real-world scenes from photos. It starts with a sparse point cloud from camera calibration and grows it into a field of 3D Gaussians, each with anisotropic covariance and spherical-harmonic color. A custom CUDA-accelerated optimizer and an OpenGL real-time viewer let you train and then freely fly through the scene at 1080p and 30 fps.

The interesting bit

The trick is abandoning neural networks entirely. Instead of querying an MLP for every ray, the scene is literally a cloud of fuzzy ellipsoids that you splat onto the screen; the renderer culls invisible Gaussians and projects the rest anisotropically, which is how it hits real-time speeds without sacrificing the visual quality normally reserved for slower radiance-field methods.

Key highlights

• Reference implementation of the ACM TOG 2023 paper, shipping with pre-trained models and evaluation datasets.
• Real-time OpenGL viewer and a network-connected training visualizer for monitoring optimization.
• Custom CUDA kernels handle anisotropic splatting and interleaved density control during training.
• Recent updates add anti-aliasing, depth regularization, exposure compensation, and OpenXR VR support.
• Requires a modern NVIDIA GPU with Compute Capability 7.0+ and 24 GB VRAM for full paper-quality training.

Caveats

• The authors explicitly warn that maintenance resources are limited, so expect sporadic updates.
• Windows builds are fragile: submodule compilation can fail with missing cl.exe, and the toolchain is picky about matching CUDA SDK and PyTorch versions.
• Training to the paper’s quality bar demands 24 GB of VRAM, pushing most consumer cards toward the FAQ for compromises.

Verdict

Worth cloning if you are reproducing radiance-field research or need a real-time baseline that trades neural-network overhead for explicit 3D Gaussians. Avoid it if you expect a plug-and-play consumer tool, active support, or own a GPU with significantly less than 24 GB of VRAM.