An LSTM that cleans speech in real time on a Raspberry Pi

What it does DTLN is a stacked dual-signal transformation LSTM network that suppresses noise from speech in real time—one audio frame in, one clean frame out. It combines short-time Fourier transform magnitude spectra with a learned analysis and synthesis basis to recover phase information, all in under one million parameters. The repository provides training and inference code alongside pretrained weights in SavedModel, TF-Lite, and ONNX formats.

The interesting bit Instead of betting everything on a single transform, the model stacks an STFT path with a learned feature basis path, letting the LSTM extract what it needs from each. That architectural hedge is what lets a sub-million-parameter network place eighth in the Interspeech 2020 DNS-Challenge real-time track while still running comfortably on a Raspberry Pi 3 B+.

Key highlights

• Runs real-time inference on a Raspberry Pi 3 B+ using quantized TF-Lite (2.2 ms per block, well under the 8 ms real-time threshold).
• Ships with pretrained models in three formats: SavedModel, TF-Lite (including quantized), and ONNX, all with external state handling for the two-part network.
• Trained on 500 hours of noisy speech from the DNS-Challenge; the author notes it can also be trained on just 40 hours with data augmentation.
• Normalized log-magnitude STFT inputs make the model insensitive to level variations without hurting objective scores.
• Outperformed the DNS-Challenge baseline by 0.24 absolute MOS points on the non-reverberant test set.

Caveats

• TF-Lite conversion and quantization shave a small amount of objective performance and add quantization noise, though the author notes the result still sounds good and hits real-time targets on a Raspberry Pi 3 B+.
• Deploying the TF-Lite or ONNX exports means juggling two separate model files and external LSTM state, which adds integration complexity compared to a single SavedModel.

Verdict Worth a look if you need deployable, low-latency speech enhancement for embedded devices or want a solid baseline for noise-suppression research. Skip it if you prioritize raw audio quality over latency and model size.