Turning sparse street views into dense voxel-splat cities

What it does SCube is a NeurIPS 2024 research pipeline that reconstructs large-scale outdoor scenes—specifically Waymo driving data—by generating dense voxel grids and then wrapping them in 3D Gaussian splats. It ingests camera and LiDAR inputs, compresses geometry through coarse-to-fine VAE stages, infers structure with latent diffusion models conditioned on images and semantics, and finally paints everything with a dedicated Gaussian Splatting Model. The result is a navigable 3D scene that can be inspected as voxel grids or raw Gaussians.

The interesting bit Instead of treating reconstruction as classical multi-view stereo, the project frames it as generative modeling, stacking a VAE, two-stage diffusion, and a splatting network into a cascading pipeline. The repository also bundles XCube, suggesting NVIDIA’s Toronto lab is quietly building an ecosystem around sparse-voxel scene generation.

Key highlights

• NeurIPS 2024 work from NVIDIA’s Toronto AI lab
• Three-stage pipeline: VAE compression, latent voxel diffusion for geometry, and a Gaussian Splatting Model for appearance
• Built around the Waymo perception dataset v1.4.2 with author-provided splits
• XCube integrated into the same codebase for easier use
• Outputs can be visualized as voxel grids, point clouds, or exported 3D Gaussian splats

Caveats

• Data preparation is brutally expensive: processing 1,000 clips takes over a day on eight A100 GPUs
• Training and inference scripts are tightly coupled to WandB for experiment tracking
• README only documents Waymo support; generalization to other datasets is unclear
• Known dependency landmines include MMCV version conflicts and Metric3D v2 runtime errors

Verdict Worth exploring if you research neural scene reconstruction, generative world models, or autonomous-driving simulation. Skip it if you need a lightweight, general-purpose 3D reconstruction tool—this is a heavy, dataset-specific research artifact.