researchmm/MM-Diffusion
A PyTorch implementation of a diffusion model that generates aligned audio-video pairs using a sequential multi-modal U-Net with separate audio and video subnets.
Not currently ranked — collecting fresh signals.
star history
This repository implements the MM-Diffusion framework for joint audio and video generation, accepted at CVPR 2023. It uses a sequential multi-modal U-Net architecture where two subnets learn to generate aligned audio-video pairs from Gaussian noise. The model supports conditional generation and was trained on datasets including landscape, AIST++, and AudioSet.
Frequently asked
- What is researchmm/MM-Diffusion?
- A PyTorch implementation of a diffusion model that generates aligned audio-video pairs using a sequential multi-modal U-Net with separate audio and video subnets.
- Is MM-Diffusion open source?
- Yes — researchmm/MM-Diffusion is open source, released under the MIT license.
- What language is MM-Diffusion written in?
- researchmm/MM-Diffusion is primarily written in Python.
- How popular is MM-Diffusion?
- researchmm/MM-Diffusion has 454 stars on GitHub.
- Where can I find MM-Diffusion?
- researchmm/MM-Diffusion is on GitHub at https://github.com/researchmm/MM-Diffusion.