wvangansbeke/Sparse-Depth-Completion
Filling in the blanks when your LiDAR is half asleep
A 2019 KITTI-topping approach that fuses sparse, noisy LiDAR with RGB images to produce dense depth maps, using learned confidence to decide which sensor to trust.
Velocity · 7d
+0.2
★ / day
Trend
→steady
star history
What it does
Takes sparse, irregular LiDAR point clouds and RGB camera images, then outputs a dense depth map for every pixel. The trick is handling the uncertainty: LiDAR gives precise depth but only where it hits, while RGB suggests structure everywhere but with no absolute scale. The network runs two branches—one for global scene understanding from RGB, one for local depth refinement—and learns confidence maps for each so it can weight their contributions.
The interesting bit
The confidence maps are learned unsupervised, which means the network figures out on its own when to trust the RGB-guided global guess versus the LiDAR-guided local correction. The author later added skip connections between the branches, which stabilized training and made the loss function choice less finicky—though this changed how the confidence maps look since fusion now happens at multiple stages rather than a single late multiplication.
Key highlights
- Ranked 1st on KITTI depth completion benchmark at submission time without extra data or post-processing
- Built on ERFNet architecture; includes a model pretrained on Cityscapes for the global branch
- RMSE of ~802 mm on KITTI selected validation set (per paper)
- Requires Python 3.7, PyTorch 1.1; preprocessing script included to downsample LiDAR and convert PNGs to JPEGs
- Creative Commons BY-NC license—research and personal use only, commercial requires contacting authors
Caveats
- The
download_raw_files.shscript comes with a warning: “at your own risk,” and the author suggests understanding it or going to KITTI’s website directly - Test evaluation may require recompiling KITTI’s provided C files for your architecture
- The training modifications (skip connections, multi-stage fusion) postdate the published paper, so results and confidence map appearance differ from the original method
Verdict
Worth a look if you’re working on sensor fusion for autonomous driving or depth completion specifically. Skip it if you need a production-ready, maintained pipeline—this is research code from 2019 with no recent commits mentioned, and the license blocks commercial use without negotiation.