Separating pop music the hard way: one LSTM per instrument

What it does

Open-Unmix is a music source separation system that takes a mixed pop track and splits it into four stems: vocals, drums, bass, and everything else. It ships as a PyTorch implementation with several pre-trained models, including a default umxl model trained on extra private data and older variants tied to the MUSDB18 dataset. Under the hood, it learns to predict magnitude spectrograms by applying a mask in the frequency domain, then wraps the separated outputs in a differentiable multichannel Wiener filter before converting back to waveforms.

The interesting bit

Instead of one giant network handling all instruments, Open-Unmix trains a separate three-layer bidirectional LSTM for each target source. While training multiple LSTMs is undeniably more work, it grants per-stem flexibility and keeps the architecture modular. The inference-time Separator then stitches these individual models together and applies a parameter-free Wiener filter to clean up the results.

Key highlights

• Ships with ready-to-use pre-trained models (umxl, umxhq, umx, and a Sony-provided speech enhancement model umxse).
• Each source model is a standalone bidirectional LSTM that operates on standardized magnitude spectrograms with learned frequency compression.
• The Separator combines outputs via a differentiable generalized Wiener filter, replacing the original norbert dependency.
• Updated to support PyTorch 2.0 as of April 2024, though the maintainers note that regression tests were relaxed and outputs may differ slightly from earlier versions.
• Can be loaded via torch.hub without installing the package, and supports both time-domain and spectrogram-domain inputs.

Caveats

• The default umxl model weights are licensed for non-commercial use only (CC BY-NC-SA 4.0).
• Because the core LSTMs are bidirectional, the system requires future audio context and cannot run in real time or online; a uni-directional variant would need separate training.
• The project is explicitly a reference implementation and inference toolkit—training scripts are not included in the PyPI package and live in separate documentation.

Verdict

Researchers benchmarking against SiSEC 2018 or audio engineers who need a quick, reproducible stem separator should grab this. If you need real-time streaming separation or a single end-to-end model to fine-tune, look elsewhere.