Whisper, diarization, and TTS — all running on your iPhone

What it does

This is a Swift package bundling three on-device speech frameworks: WhisperKit (speech-to-text via OpenAI Whisper), SpeakerKit (speaker diarization via Pyannote), and TTSKit (text-to-speech via Qwen-TTS). Everything runs locally on Apple Silicon — no cloud round-trips, no API keys, no per-minute billing. Models are CoreML-converted and pulled from Argmax’s HuggingFace repos, with a recommended default per device if you don’t want to think about it.

The interesting bit

The bundled CLI ships a local server that implements the OpenAI Audio API — so your existing OpenAI SDK clients can point at localhost and transcribe against hardware you already own. There’s also a companion repo, whisperkittools, for converting your own fine-tuned Whisper models to CoreML.

Key highlights

• Three independent library products in one package; grab just the kit you need or the ArgmaxOSS umbrella
• Local server speaks the OpenAI Audio API, including streaming transcription results
• WhisperKit supports model selection by name (with glob matching) and custom model repos
• TTSKit includes custom voices, real-time streaming playback, and style instructions on the 1.7B model
• SpeakerKit outputs RTTM and can be combined with transcription for labeled speaker turns

Caveats

• Apple-only: requires macOS 14+ / Xcode 16+; Android and non-native app support live in the paid Pro SDK
• Real-time transcription with speaker labels is a Pro feature, not in this repo
• The README is openly a funnel — the Pro SDK gets prominent placement at the top

Verdict

If you’re building an Apple-native app that needs private, offline speech processing, this is one of the most complete open-source options around. If you’re not in the Swift/Xcode world, the free tier mostly isn’t for you.