An AI Foley Artist That Actually Hits Its Cues

What it does

FoleyCrafter is a video-to-audio generator that watches silent clips and fabricates matching sound effects—footsteps, impacts, ambient noise—from scratch. It builds on the Auffusion text-to-audio model, adding video-specific adapters so the output is semantically relevant and temporally aligned with what happens on screen. You can also steer generation with text prompts when the visuals alone are ambiguous.

The interesting bit

The system separates the problem into what sound and when it happens: a semantic adapter picks plausible audio concepts, while a temporal adapter and timestamp detector lock those events to specific visual cues. That keeps a door slam from drifting two seconds away from the impact.

Key highlights

• Diffusion-based generation built atop the Auffusion text-to-audio base model
• Temporal alignment module with dedicated adapter and timestamp detector to sync audio to visual events
• Optional text prompts for semantic steering beyond pure video context
• Gradio demo and HuggingFace Space available for browser-based experimentation
• Inference code and checkpoints released; training code is still pending according to the roadmap

Caveats

• Training code has not yet been released, so fine-tuning or retraining from scratch isn’t currently possible
• The pipeline depends on several specialized checkpoints—semantic adapter, temporal adapter, vocoder, and timestamp detector—in addition to the base model, so it is not a lightweight single-file deployment

Verdict

Video editors, animators, and generative-media tinkerers who need plausible audio for mute footage should take a look. Researchers hunting for a fully open, trainable pipeline will need to wait for the training code release.