An independent-ish shootout of Korean speech-to-text APIs

What it does

This repository gathers a field of ten Korean speech-to-text APIs—Whisper, Google, Azure, AWS, Naver, ETRI, and others—and scores them with real-world error rates. It runs each service on roughly 3,000 samples from six AI-Hub public datasets, spanning noisy meetings, phone calls, and lectures, then ranks them by Character Error Rate. The result is a public scoreboard that shows which APIs keep up with casual Korean speech and which fall behind.

The interesting bit

The maintainers are Return Zero, a Korean STT vendor, and their own API tops the table with a 5.91% average CER—ahead of Naver ClovaSpeech at 7.52%. Rather than bury that conflict, they publish the methodology, normalization rules, and a fine-tuned Whisper variant, turning a marketing exercise into something approaching engineering transparency.

Key highlights

• Ten APIs tested across six Korean datasets: meetings, consultations, low-quality phone audio, lectures, and conversational speech.
• Uses Character Error Rate instead of Word Error Rate, because Korean word boundaries are ambiguous and morphologically complex.
• Return Zero’s own API leads the pack; its fine-tuned Whisper variant places second among tested services.
• Google Cloud Speech-to-Text v2 failed one dataset entirely due to a 10 MB / 60-second file-size limit.
• Includes a short explainer on why CER matters for agglutinative languages, with worked examples showing how WER punishes harmless spacing differences.

Caveats

• The benchmark is run by Return Zero, who also happen to win it; independent replication would strengthen the claims.
• Each test set is sampled down to 3,000 sentences for cost and time reasons, so edge-case performance may differ from full-dataset results.
• Several major providers—Amazon Transcribe, Microsoft Speech Service, and Kakao—are listed as planned but not yet tested.

Verdict

Worth bookmarking if you are choosing a Korean STT provider or arguing about evaluation metrics. Skip it if you need English-only benchmarks or fully vendor-neutral audits.