Sync AI video to drum hits inside ComfyUI

What it does This is a set of custom nodes for ComfyUI that analyze an audio track—separating it into drums, vocals, bass, and other stems—then generate frame-by-frame weights and peak triggers. Those signals drive existing ComfyUI tools like IPAdapter, AnimateDiff, and ControlNet so your generated animations actually hit on the beat instead of floating in silence.

The interesting bit Rather than treating audio as a single waveform, it uses stem separation models (HybridDemucs or OpenUnmix) to let you, say, trigger a style transition only when the vocals kick in, or remap prompts on every snare hit. It turns ComfyUI’s node graph into a DAW-style automation lane for video generation.

Key highlights

• Built-in stem separation and audio remixing (drums, vocals, bass, others) with visual waveform previews
• Peak detection outputs binary triggers, indices, and graphs you can wire directly into IPAdapter transitions or prompt schedulers
• Drop-in JSON workflows for three patterns: images-to-video, video-to-video, and text-to-video
• Plays nice with the existing ComfyUI ecosystem without replacing your pipeline
• Utility nodes for float inversion, mask-to-float conversion, and weight-strategy formatting to bridge audio math into image-model parameters

Caveats

• Requires an existing ComfyUI setup; this is an extension, not a standalone tool
• Several included nodes (Invert Floats, Repeat Image To Count) are basic glue rather than audio-specific logic
• The README shows results and tutorials but stays quiet on hardware requirements, processing speed, or maximum track length

Verdict Grab this if you already live in ComfyUI and want your generative videos to feel choreographed rather than randomized. If you are looking for a self-contained audio-reactive video editor, keep looking.