Swap skies in video without motion sensors or manual rotoscoping

What it does SkyAR ingests a video and composites a new sky behind the foreground—floating castles, supermoons, or anything else—while attempting to match lighting and motion dynamics. It segments the sky, estimates background motion, and blends the replacement layer into the original footage. Because the approach is entirely vision-based, it works on any existing video from dash cams or smartphones without requiring special capture hardware such as inertial measurement units.

The interesting bit The authors break the problem into three proxy tasks—sky matting, motion estimation, and image blending—rather than attacking it as one end-to-end black box. This decomposition is what lets the same pretrained model generalize across diverse in-the-wild footage in real time with no user interaction, a combination that usually demands green screens or painstaking per-frame rotoscoping.

Key highlights

• Purely vision-based pipeline: no IMU or capture-side hardware dependencies
• Real-time inference with tunable relighting, recoloring, and halo effects
• Ships with a pretrained sky matting model and full training code (though the included dataset is only a small structural sample)
• Handles both online streams and offline video files
• Non-commercial license (CC BY-NC-SA 4.0)

Caveats

• The sky matting model is trained solely on daytime images, so it frequently misdetects sky regions in nighttime footage
• When the sky leaves the frame or lacks texture, motion estimation falters and the replacement background drifts out of sync with the foreground

Verdict Researchers and visual-effects hobbyists looking for an automatic, hardware-agnostic sky swap for daylight video will find this immediately useful. Anyone working primarily with night scenes or heavily occluded horizons should wait for a retrained model.