One RGB image, any lens: metric 3D from pinhole to panoramic

What it does

UniK3D takes a single RGB image and outputs a metric 3D point cloud, depth map, and ray directions. It handles standard pinhole cameras but also natively supports fisheye, wide-angle, and spherical panoramic lenses, and it does not require known intrinsics—though it will use them if provided. The project provides pre-trained ViT backbones, a local Gradio demo, and a Hugging Face space for quick testing.

The interesting bit

Instead of hard-coding a pinhole projection, the model predicts ray directions and metric depth, letting it reconstruct scenes from any camera geometry. The zero-shot evaluation numbers are lopsided: on panoramic images it hits 80.2 F1 while competitors barely break 3.7.

Key highlights

• Supports arbitrary camera models—Pinhole, Fisheye624, MEI, and Spherical—through a unified ray-based representation
• Predicts metric 3D point clouds, depth, and unprojected rays from a single image, with or without known camera parameters
• Zero-shot F1 scores on panoramic and large-FoV benchmarks are roughly 20–40× higher than UniDepth, MASt3R, and DepthPro
• Three model sizes (ViT-S/B/L) on Hugging Face and TorchHub, plus a browser Gradio demo
• CVPR 2025; training and inference code are released

Caveats

• Licensed under CC BY-NC 4.0, so commercial use is off the table
• Linux and CUDA are effectively required; macOS and Windows are untested
• “Rays to parameters optimization” remains on the maintainers’ unchecked todo list

Verdict

Grab it if you work with mixed or exotic camera rigs in robotics or computer vision. Skip it if you need a commercially deployable drop-in for standard pinhole depth estimation.