The Crate-Digger’s ML Pipeline: From MP3s to Stems and MIDI

What it does Polymath ingests audio from local files or YouTube and runs it through a pipeline of existing ML models. It separates tracks into stems like bass and drums, detects key and structure, quantizes everything to a shared beat grid, and spits out a searchable library. The output is meant to drop straight into a DAW or feed into a generative model training run.

The interesting bit Rather than training its own models, it acts as a disciplined stage manager for Demucs, Crepe, Basic Pitch, and others—gluing them together so producers and ML developers don’t have to wrangle six separate tools just to isolate a breakbeat or build a dataset. The payoff is bulk automation: point it at a folder and it returns stems, MIDI, and metadata in a uniform format.

Key highlights

• Stems out six parts (bass, drums, guitar, piano, vocals, other) via Demucs.
• Quantizes and beat-locks tracks to a common tempo using pyrubberband.
• Detects musical structure (verse, chorus), key, pitch, timbre, and loudness.
• Converts audio to MIDI with Spotify’s Basic Pitch model.
• Searches for similar tracks across the library to help build mixes or datasets.

Caveats

• Generated MIDI files are hard-locked to 120 BPM for now and need manual timing adjustments in your DAW.
• The audio-to-MIDI model struggles with drums and percussion, so don’t expect clean kit transcriptions yet.
• Docker GPU support is listed as “forthcoming,” so native installs are the only path for CUDA acceleration today.

Verdict Worth a look if you are a producer, DJ, or ML audio researcher sitting on a mountain of unorganized audio. Skip it if you need a polished, end-to-end DAW plugin or flawless MIDI transcription today.