Whisper, but it actually writes down every 'um' and 'uh'

What it does

CrisperWhisper is an advanced variant of OpenAI’s Whisper built for verbatim transcription. It outputs word-level timestamps and explicitly transcribes disfluencies—fillers like "um" and "uh", stutters, pauses, and false starts—instead of smoothing them into clean prose. It also claims to reduce hallucinations relative to the original model.

The interesting bit

The authors adjusted the tokenizer and added a custom attention-loss term during training to force the model to learn sharper alignments around hesitations. That extra loss specifically targets the attention scores used for DTW-based timestamp alignment, which is exactly the kind of boring detail that makes the timestamps actually crisp.

Key highlights

• Ranks first on the Hugging Face OpenASR Leaderboard for verbatim datasets (TED, AMI) and overall.
• Beats Whisper Large v3 on AMI (8.72 vs 16.01 WER), LibriSpeech clean, SPGISpeech, TED-LIUM, VoxPopuli, and CommonVoice; roughly on par or slightly behind on Earnings22, GigaSpeech, and LibriSpeech other.
• Segmentation F1 and IOU scores exceed Whisper Large v2/v3 on AMI IHM, Common Voice, and TIMIT, especially around disfluencies.
• Accepted at INTERSPEECH 2024; the paper covers tokenizer adjustments, while the attention-loss improvement was added after publication.
• Requires the authors’ custom transformers fork for the most accurate timestamps; a faster-whisper conversion exists but the README warns timestamp accuracy is not guaranteed there.

Caveats

• The faster-whisper compatible model sacrifices timestamp accuracy because CTranslate2 calculates timestamps differently.
• You must use the authors’ custom transformers fork rather than the upstream library to get the full timestamp precision.

Verdict

Worth a look if you need verbatim transcripts that preserve every filler and false start for analysis or archival purposes. Skip it if you just want clean, readable captions and are happy with standard Whisper’s editorializing.