Inactive pipeline that generates music by hallucinating spectrograms

What it does Riffusion is a Python library that repurposes Stable Diffusion to generate music and audio. It generates and interpolates spectrogram images using a diffusion pipeline, then converts those images into actual audio clips. The repo includes a Streamlit playground, a Flask inference server, and command-line utilities for image-to-audio tasks.

The interesting bit The whole trick is treating frequency plots as paintings. Because the model operates on spectrograms as if they were ordinary images, you can use standard image diffusion techniques—like prompt interpolation and image conditioning—to steer audio generation without designing a new neural architecture.

Key highlights

• Generates audio by running Stable Diffusion on spectrogram images
• Supports prompt interpolation with image conditioning for sound transitions
• Ships with a Streamlit UI, Flask API server, and CLI tools
• Real-time generation requires a high-end GPU such as a 3090 or A10G
• Explicitly no longer actively maintained

Caveats

• Project is no longer actively maintained
• CPU inference is “quite slow”; Apple Silicon MPS falls back to CPU for audio processing and is non-deterministic
• Tested only with diffusers 0.9–0.11 and Python 3.9/3.10

Verdict Worth exploring if you want to see how an image model can be pressed into service as a music generator, or if you need local spectrogram-to-audio utilities. Look elsewhere for a maintained, production-ready audio synthesis pipeline.