Auto-generate karaoke game charts from any song with AI

What it does

UltraSinger ingests audio or video—including YouTube links—and runs it through an AI pipeline that separates vocals, transcribes lyrics with word-level timing, detects pitch, and emits finished UltraStar karaoke charts plus MIDI and sheet music. It can also take an existing UltraStar file and re-pitch it against a new audio source. The goal is eliminating the manual drudgery of building playable singing-game tracks.

The interesting bit

The project chains together specialized models—Whisper for lyrics, SwiftF0 (faster and more accurate than CREPE) for pitch, and demucs for stem separation—then quantizes detected notes to the song’s musical key to scrub out pitch slides and off-key errors. It even estimates the in-game score you’d likely receive, which is a tidy feedback loop for a chart generator.

Key highlights

• Accepts local audio/video files or YouTube URLs
• Separates vocals, transcribes with Whisper, and detects pitch with SwiftF0
• Exports UltraStar.txt, MIDI, and sheet music (requires local MuseScore)
• Re-pitches existing UltraStar files against new audio
• Calculates theoretical in-game scores for generated charts

Caveats

• Explicitly marked as under development; several CLI features are listed as “in progress”
• GPU users may hit VRAM crashes and need to dial down batch sizes or switch to CPU
• Sheet music generation depends on a locally installed MuseScore executable

Verdict

Party hosts and karaoke die-hards looking to bulk up their UltraStar library will get the most mileage. Anyone needing polished, commercial-grade charts should probably wait for the rough edges to smooth out—or stick to manual editing.