Yeah, It Sings: Open-Source Music Generation for the Patient

What it does

YuE is a foundation model that takes lyrics and genre prompts and generates complete songs several minutes long, complete with separate vocal and instrumental tracks. It handles multiple languages and genres, and can clone a reference style or voice if you feed it existing audio. The project is explicitly positioned as an open alternative to Suno.ai, down to the Apache 2.0 license that lets artists sample and monetize outputs.

The interesting bit

The pipeline chains a 7B stage-one checkpoint, a 1B stage-two checkpoint, and an upsampler, using inference flags like max_new_tokens and repetition_penalty that feel borrowed from text-generation stacks. Its in-context learning mode is the real curiosity: feed it 30 seconds of split vocal and instrumental tracks, and it writes a new song in that exact style, turning the system into an audio style-transfer engine.

Key highlights

• Generates full songs with distinct vocal and accompaniment tracks from text prompts and lyrics.
• Supports multiple languages (English, Japanese/Korean, Chinese) with specialized model variants.
• Dual-track in-context learning enables voice cloning and style transfer from reference audio.
• Apache 2.0 license explicitly encourages commercial use and sampling by artists.
• Community forks like YuEGP and YuE-exllamav2 squeeze the model onto 24 GB GPUs, though the authors warn this may compromise musicality.

Caveats

• Full song generation requires roughly 80 GB of VRAM (H800, A100, or multi-GPU setups); 24 GB cards are limited to short two-session snippets.
• Inference is slow: 30 seconds of audio takes 150 seconds on an H800 and about six minutes on an RTX 4090.
• Several promised optimizations—llama.cpp support, vLLM/sglang serving, tensor parallelism—remain on the TODO list.

Verdict

Musicians and researchers who want an open-weight alternative to closed AI music generators should look here, provided they have the hardware or patience for community quantization work. Casual hobbyists with consumer GPUs will likely find the experience closer to watching paint dry than to instant songwriting.