A speech pipeline that treats Chinese dialects as first-class citizens

What it does FireRedASR2S is an integrated speech-processing system that packages four modules: an ASR engine (FireRedASR2), a voice-activity detector (FireRedVAD), a spoken-language identifier (FireRedLID), and a punctuation restorer (FireRedPunc). The ASR module handles Mandarin, over 20 Chinese dialects and accents, English, and code-switched audio—including both speech and singing transcription. Each module ships with published benchmark numbers claiming state-of-the-art results against competitors like Doubao-ASR, Qwen3-ASR, FunASR, Silero-VAD, Whisper, and SpeechBrain.

The interesting bit The project offers two distinct ASR architectures under one umbrella: an LLM-backed variant (FireRedASR2-LLM) for end-to-end interaction and a lighter Attention-based Encoder-Decoder (FireRedASR2-AED) that can double as a speech representation module for larger pipelines. The VAD module also moonlights as a lightweight audio-event detector via multi-label VAD, tagging speech, singing, and music simultaneously.

Key highlights

• Claims 2.89% CER on Mandarin benchmarks and 11.55% on 19 Chinese dialect/accent test sets, ahead of Doubao-ASR and Qwen3-ASR-1.7B.
• VAD supports 100+ languages and scores a 97.57% F1 on a 102-language benchmark, while LID reports 97.18% accuracy on FLEURS.
• FireRedASR2-AED supports word-level timestamps and confidence scores; the LLM variant can run inside vLLM.
• NVIDIA contributed TensorRT-LLM acceleration for the AED model, claiming a 12.7× speedup on an H20 GPU over PyTorch.
• Punctuation prediction covers Chinese and English with a reported 78.90% average F1.

Caveats

• The README defers most architectural detail to the arXiv technical report, so you will need the paper for design rationale.
• Fine-tuning code and the FLEURS-VAD-102 dataset are both listed as “coming soon,” which limits reproducibility to inference for now.

Verdict Worth a look if you are building Chinese-centric voice applications that need to juggle dialects, code-switching, or singing transcription. Skip it if you only need generic English ASR and do not want to wrangle a multi-model pipeline.