Strip PyTorch from your speech recognizer, keep the accuracy

What it does

This is a standalone C++ inference engine for speech recognition. It takes pretrained acoustic models—sourced from PaddleSpeech, Kaldi2, and Alibaba’s Paraformer—and runs them without installing PyTorch, PaddlePaddle, or TensorFlow. The project ships both a static C++ library and a Python module, and the README claims accuracy comparable to commercial ASR engines on Chinese (and some English) speech.

The interesting bit

Instead of wrapping existing frameworks, the author rewrote the inference graph in raw C++, using aggressive pointer arithmetic to eliminate redundant reshapes and memory copies. Pretrained weights must be converted from their native formats into a custom binary layout, after which the engine can memory-map them directly.

Key highlights

• Runs on ARM: the author specifically targets the Raspberry Pi 4B and similar low-power boards.
• Lean dependency tree: only libfftw3 and libopenblas are required beyond a standard C++ toolchain.
• Four model variants: three non-streaming (sentence-level, higher accuracy) and one streaming conformer_online model that trades accuracy for latency.
• Non-streaming models support long-form transcription via VAD.
• Inference timing in the README shows a 5-second utterance processed in roughly 0.6–0.7 seconds, though the test hardware is unspecified.

Caveats

• Quantization and model compression are listed as unfinished work.
• The streaming conformer_online model explicitly sacrifices some accuracy for real-time response.
• Pretrained weights must be converted from PyTorch or PaddlePaddle checkpoints into a custom binary format before use.

Verdict

Worth a look if you need Mandarin speech-to-text on embedded Linux or x86 without the usual framework bloat. Skip it if you require production-ready streaming with top-tier accuracy or broad English support beyond the bilingual Paraformer model.