Real-time Whisper as a server, with swappable silicon backends

What it does

WhisperLive is a server-client real-time transcription system built around OpenAI’s Whisper model. It accepts live audio from microphones, pre-recorded files, or network streams like RTSP and HLS, and returns partial transcripts over a WebSocket as the audio plays. The server can also translate incoming speech to English on the fly and optionally tag segments by speaker.

The interesting bit

Rather than hard-coding one inference stack, the server abstracts over three backends—faster_whisper, NVIDIA TensorRT-LLM, and OpenVINO—so you can optimize for the silicon you actually have. It also batches multiple client sessions into single GPU calls and exposes extras like per-word timestamps and custom hotwords, which are usually afterthoughts in open-source speech pipelines.

Key highlights

• Three inference backends—faster_whisper, TensorRT-LLM, and OpenVINO—covering NVIDIA GPUs, Intel CPUs, and Intel iGPUs/dGPUs.
• Optional speaker diarization via pyannote.audio with online cosine-similarity clustering.
• Word-level timestamps and per-word probability scores.
• Custom vocabulary boosting via hotwords passed directly to the backend.
• Batch inference for grouping multiple client sessions into one GPU call, plus browser extensions and raw PCM support for embedded devices.

Caveats

• The TensorRT backend is currently recommended only inside Docker and may emit repeated CrossAttentionMask warnings or crash unless you switch to a Python session.
• OpenVINO GPU support is easiest in Docker; native use requires manual Intel driver and runtime setup.
• Speaker diarization depends on an optional pyannote.audio dependency; if missing, the server logs a warning and continues without identifying speakers.

Verdict

A solid choice if you need a self-hosted, low-latency transcription pipeline with hardware flexibility and production niceties. Look elsewhere if you want a fully managed API without server administration.