A Swiss-army diffusion library for audio, batteries not included

What it does

audio-diffusion-pytorch is a PyTorch library for building diffusion models that operate directly on audio waveforms. It bundles several architectures—unconditional and text-conditional generators, upsamplers, vocoders, autoencoders, and an inpainter—around a generic U-Net backbone and a V-parameterization diffusion sampler. Despite the audio focus, the core DiffusionModel, U-Net, and samplers are dimension-agnostic, so they can be repurposed for other data formats.

The interesting bit

The library treats diffusion as a configurable pipeline rather than a monolithic model. You plug in your own encoder for the autoencoder, swap samplers, or layer cross-attention for T5 text conditioning, all while staying in the time domain rather than hiding inside a spectrogram. The README is admirably blunt that the example configs are “indicative and untested,” pointing you to the Moûsai paper for battle-tested settings.

Key highlights

• Waveform-based generation, upsampling (e.g., 3 kHz to 48 kHz), vocoding from mel-spectrograms, and latent autoencoding with custom encoders.
• Text-to-audio conditioning via T5 embeddings with built-in classifier-free guidance.
• Dimension-agnostic U-Net (a-unet), diffusion method, and samplers that can be adapted beyond audio.
• Includes an inpainting module (VInpainter) for masked audio completion.
• ~2,100 stars and ties back to the Moûsai research paper for proven configurations.

Caveats

• No pre-trained checkpoints are provided; everything must be trained from scratch.
• The example configurations in the README are explicitly labeled “indicative and untested.”
• Text conditioning requires an external transformers dependency for T5.

Verdict

Grab this if you are a researcher or tinkerer who wants a clean, hackable diffusion substrate for audio experiments and already has a GPU budget for training. If you are looking for a ready-to-run Stable Diffusion equivalent for sound, keep looking.