An open-source audio model with a sense of time

What it does

MOSS-Audio is a family of 4B and 8B audio-language models built on Qwen3. Feed it audio and it will transcribe, caption, answer questions, or reason about speech, music, and environmental noise. The release includes both direct-instruction variants and chain-of-thought “Thinking” variants trained with reinforcement learning.

The interesting bit

The team skipped off-the-shelf audio frontends and trained a dedicated encoder from scratch, then wired it to the LLM using a DeepStack-style cross-layer injection that pushes low-level acoustic features—rhythm, timbre, transients—into the language model’s early layers. They also insert explicit time tokens during pretraining so the model learns “what happened when,” which apparently pays off: the 8B-Instruct model scores 35.77 AAS on AISHELL-1 timestamp ASR, next to Qwen3-Omni’s 833.66.

Key highlights

• Four released sizes: 4B and 8B, each in Instruct and Thinking (chain-of-thought) flavors
• Custom encoder running at 12.5 Hz, not an off-the-shelf whisper clone
• Cross-layer feature injection preserves prosody and transient details that top-layer features usually discard
• Time-marker insertion enables word-level and sentence-level timestamp alignment inside a standard text-generation framework
• Claims lowest overall CER (11.30) across a 12-dimension ASR benchmark suite and leads 11 of 13 speech-captioning dimensions

Caveats

• The README is heavy on benchmark tables but light on training data details or hardware requirements
• Benchmark emphasis varies by task: speech captioning and timestamp ASR highlight Instruct variants, while general audio understanding scores favor the Thinking models

Verdict

Worth a look if you need open, unified audio understanding with precise timestamps. Skip it if you want a managed speech-to-text API—this is a research release with model weights and fine-tuning scripts, not a plug-and-play service.