Video matting that remembers the previous frame

What it does RVM separates humans from video backgrounds in real time, outputting an alpha matte and foreground for each frame. It is built around a recurrent neural network that carries memory across frames, and it requires no extra inputs like trimaps or background plates. The project ships pretrained weights for MobileNetv3 and ResNet50 backbones, with exports to PyTorch, ONNX, TensorFlow, TensorFlow.js, CoreML, and TorchScript.

The interesting bit Most neural matting models treat each frame as an isolated image, which tends to flicker. RVM passes recurrent states forward so the network knows what the subject looked like a moment ago. The authors claim this architecture hits 4K 76 FPS and HD 104 FPS on a GTX 1080 Ti, and up to 172 FPS at HD on an RTX 3090.

Key highlights

• Runs fully offline on arbitrary video without additional inputs
• Exported to six inference frameworks, including browser-ready TensorFlow.js and mobile CoreML
• Includes a live browser webcam demo that visualizes the recurrent states
• Two model sizes: lightweight MobileNetv3 and a heavier ResNet50 variant
• Originated as a WACV 2022 paper from ByteDance

Caveats

• The published speed figures are pure tensor throughput; the bundled video conversion script is expected to be much slower because it lacks hardware video encoding/decoding and parallel tensor transfer
• CoreML exports are locked to fixed resolutions and require iOS 13+

Verdict Grab it if you need real-time human segmentation for video calls, editing pipelines, or browser apps. Look elsewhere if you need generic object matting—RVM is specifically tuned for humans.