Teaching neural nets to hallucinate depth for 3D movies

What it does Deep3D takes ordinary 2D video and generates a stereoscopic pair—left-eye and right-eye views—so you can watch it on a 3D display or VR headset. It does this by estimating how far each pixel is from the camera, then warping the image to create the second viewpoint. The whole pipeline lives in a single Jupyter notebook with a pretrained model.

The interesting bit The clever part is the training data hack. Depth-labeled datasets like NYU Depth or KITTI are tiny and mostly static, so they don’t generalize to people or complex scenes. Deep3D sidesteps this by training directly on existing 3D movies—tens of millions of frames—where the “ground truth” is simply the other eye’s view. Depth becomes an internal layer the network learns to infer, not something it explicitly predicts. It’s a neat end-run around the data bottleneck.

Key highlights

• End-to-end CNN: depth estimation and view synthesis are trained jointly, not as separate stages
• Trained on 3D movie footage, avoiding scarce depth-map datasets
• Includes visualizations of internal depth layers (4×3 grids from near to far)
• Ships as a single deep3d.ipynb notebook with pretrained weights
• Built on MXNet with custom CUDA operators

Caveats

• Requires compiling MXNet from source with CUDA 7.0 and cuDNN 4, plus manual operator injection—setup is decidedly 2016-era
• Training code was promised “shortly” in the README; unclear if it ever arrived
• No stated license, update history, or community activity visible in the sources

Verdict Worth a look if you’re researching view synthesis or historical deep-learning architectures. Skip it if you need something production-ready or modern—this is a research artifact from the MXNet era with the maintenance profile to match.