SIGGRAPH paper turns diffusion models into cartoon in-betweeners

What it does

ToonCrafter takes a starting and ending cartoon frame and hallucinates the video clip that connects them. It is built on top of pre-trained image-to-video diffusion models, repurposing those priors specifically for 2D cartoon content. The system also accepts sparse sketch guidance to nudge the motion, and can colorize line-art sketches when given a reference frame.

The interesting bit

The authors openly admit the “success rate is not guaranteed” because generative video priors are fickle—a refreshing level of honesty for a research repo. Rather than training a cartoon pipeline from scratch, they leverage existing diffusion priors and bend them toward animation interpolation, which is why the project can run at all without a Hollywood budget.

Key highlights

• Generates up to 16 frames at 512×320 resolution between two input images.
• Official checkpoint needs roughly 24 GB of VRAM, though community pruned fp16 builds have cut that to about 10–12 GB.
• Supports sparse sketch guidance to influence motion paths, plus reference-based sketch colorization.
• Accepted at SIGGRAPH Asia 2024 (Journal Track / ACM TOG).
• Active community ecosystem: ComfyUI nodes, Colab notebooks, and Windows ports already exist.

Caveats

• The README lists sketch control and colorization functions as a pending todo, so those features may not be fully integrated in the main branch yet.
• It is explicitly framed as a research exploration, not a commercial tool, and output quality can be inconsistent.

Verdict

Worth a look if you are an animator, researcher, or ComfyUI tinkerer exploring generative in-betweening. If you need a reliable, production-grade tweening tool with deterministic output, this is not it.