MIMO puppeteers any portrait through novel 3D motion

What it does MIMO generates realistic character videos by animating a single input image with controllable 3D motion, whether from motion-capture data or an in-the-wild driving video. It treats character identity, motion trajectory, and background scene as independently tunable attributes inside one diffusion model.

The interesting bit The project uses spatial decomposed modeling to disentangle who is moving from how and where they move, then reassembles the pieces through a Stable Diffusion backbone. That is the architectural bet behind its claim of handling arbitrary characters and novel motions in a single framework.

Key highlights

• CVPR 2025 official PyTorch implementation from Alibaba’s Tongyi Lab
• Supports both character image animation and video character editing
• Claims unified scalability to arbitrary characters, novel 3D motions, and interactive real-world scenes
• Local Gradio demo requires ≥40 GB VRAM; tested on A100 and L20 GPUs
• Built atop Stable Diffusion 1.5 with added motion modules, pose guiders, and reference U-Nets

Caveats

• The released code is a “simplified version” of the full implementation; the README says it achieves comparable performance but does not spell out what was simplified
• Custom driving videos need preprocessing through a separate decomposition pipeline with extra dependencies and model weights
• Requires collecting several external pretrained models (Stable Diffusion V1.5, VAE, image encoder) in addition to the project’s own checkpoints

Verdict Researchers in controllable video generation should find the architecture worth studying. Casual experimenters will likely prefer the hosted ModelScope demo over assembling weight collections and meeting 40-gigabyte hardware requirements.