Self-hosted voice assistant with barge-in and 9× realtime TTS

What it does This is a real-time conversational voice assistant that runs entirely on your own hardware. You speak into a browser, it transcribes audio via faster-whisper or Moonshine, routes intent through a local LLM such as Ollama or vLLM, and streams back synthesized speech through its own 300M-parameter Vui Nano model. The whole pipeline—ASR, reasoning, and TTS—lives in a single Python server with a WebRTC frontend.

The interesting bit The project treats conversation as a real-time control problem, not a batch job. It performs speculative LLM prefill while you are still speaking, chunks TTS at the sentence level with backpressure, and supports barge-in so you can interrupt mid-reply. The TTS model was trained on conversational speech including breaths, laughter, and hesitations, and exposes conditioning knobs for speech quality and words-per-second.

Key highlights

• OpenAI Realtime API-compatible WebSocket endpoint (ws://…/v1/realtime), so existing clients can drop it in
• Vui Nano: 300M-parameter Llama-style decoder with RQ-Transformer head, ~9× realtime inference on an RTX 4090
• Pluggable backends: swap ASR (faster-whisper, Moonshine) and LLM (Ollama, vLLM, OpenAI-compatible) live from the UI
• Voice cloning from audio samples plus four shipped presets (maeve, abraham, rhian, harry)
• Optional Claude task server sidecar for agentic work (Gmail, Calendar, Slack) via MCP, with non-Anthropic fallback backends

Caveats

• Apple Silicon support via MLX is explicitly work-in-progress and lacks the polish of the CUDA path
• Docker setup is designed primarily for Linux + NVIDIA GPU; other combinations may need manual tuning
• The optional Claude task server requires separate setup; without it, agentic delegation features are unavailable

Verdict Worth a look if you want a hackable, fully local voice stack with OpenAI-compatible wiring. Skip it if you need a polished Apple Silicon experience today or a fully turnkey macOS deployment.