OpenAI’s archived music model needs three hours for 20 seconds of audio

What it does

Jukebox generates music from scratch or extends primed audio using a three-level hierarchy of VQ-VAEs and autoregressive Transformers. The released code includes pretrained models up to 5B parameters, including lyric-conditioned variants, and can upsample coarse tokens into 44.1 kHz waveforms. It also exposes training pipelines so you can retrain just the top-level prior on your own dataset while keeping the pretrained encoder and upsamplers.

The interesting bit

The project is explicitly archived—OpenAI provides it as-is with no updates expected—yet it remains a rare example of modeling raw audio waveforms directly rather than MIDI or spectrograms. The cost of that fidelity is steep: on a V100, generating 20 seconds of audio takes roughly three hours, and the 5B lyrics model consumes 11.5 GB of GPU memory for its top-level prior alone, plus another gigabyte per sample for transformer key-value cache.

Key highlights

• Generates music with lyrics alignment via a 5-billion-parameter top-level prior
• Supports primed generation from your own WAV files, not just random sampling
• Three-tier pipeline: top-level prior plans structure, two upsampler levels fill in audio detail
• Can reuse pretrained VQ-VAE and upsamplers to train a new style-specific prior from scratch
• Ships with a Colab notebook and an HTML viewer that animates lyrics alongside playback

Caveats

• Explicitly archived with no updates expected; treat it as a frozen research artifact
• Targets V100 GPUs with 16 GB of memory; smaller cards will likely struggle with the 5B model
• Sampling is serial and slow—most of the three-hour runtime is spent in upsampling

Verdict

Study it if you want to understand large-scale raw-audio generation or fine-tune a prior on a narrow musical domain. Skip it if you need a maintained, real-time, or consumer-grade music synthesis tool.