Whisper, but it actually listens in real time

What it does

Moonshine is a C-based voice AI toolkit that handles speech-to-text, text-to-speech, speaker diarization, and intent recognition entirely on-device. It ships as a single library with identical APIs across Python, iOS, Android, macOS, Linux, Windows, and Raspberry Pi — no cloud accounts, no API keys, no network required.

The interesting bit

The core trick is architectural: unlike Whisper, which insists on a fixed 30-second audio window and starts from scratch on every streaming call, Moonshine accepts variable-length input and caches encoder/decoder state incrementally. On a MacBook Pro, their Medium model returns results in 107ms versus Whisper Large v3’s 11,286ms. Even the 34MB Tiny variant runs in 34ms — faster than Whisper Tiny’s 277ms while beating its accuracy. The README is refreshingly direct about why they built this: they were “early and enthusiastic adopters of Whisper” who kept hitting walls building live voice interfaces.

Key highlights

• Models trained from scratch, not Whisper derivatives; claims higher accuracy than Whisper Large V3 on Open ASR Leaderboard at the top end, down to 26MB for edge deployments
• Streaming STT with state caching: compute happens while the user is still talking
• Language-specific models rather than one multilingual model that handles many languages poorly — Asian languages (Korean, Japanese, Mandarin) are specifically called out as Whisper weak spots
• “Batteries included” APIs cover transcription, TTS, diarization, command recognition, and conversational agents
• 15 TTS languages versus 8 STT languages; the language coverage asymmetry isn’t explained

Caveats

• The README truncates mid-sentence during the “Language-specific models” section, so details on which languages get dedicated models versus the full list remain unclear
• Benchmarks are self-reported with methodology referenced but not shown in the truncated source; the 5× speedup claim from first-generation models lacks independent verification in the visible text
• Some platform quickstarts suggest --break-system-packages for Raspberry Pi, which hints at packaging friction

Verdict

Build voice agents for constrained or privacy-sensitive environments — this is purpose-built for that. If you’re batch-processing archived audio files and don’t care about latency, Whisper’s ecosystem maturity probably still wins.