Zephyr 7B Learns to Talk, Glitches and All

What it does

This project wires together Zephyr 7B (via llama.cpp), real-time speech-to-text, and text-to-speech into a single offline voice chatbot. You speak, it transcribes with faster_whisper, thinks with a local 7B model, and replies aloud using Coqui XTTS—optionally in a voice cloned from a WAV file you provide. Personality tweaks live in a JSON file, and the whole stack stays on your GPU.

The interesting bit

The heavy lifting isn’t the model choice; it’s the latency juggling act between transcription, inference, and synthesis. The author also maintains both the RealtimeSTT and RealtimeTTS libraries, so this chatbot doubles as a dogfooding demo for a vertically integrated audio pipeline.

Key highlights

• Runs fully offline; no cloud API keys or network calls required.
• Voice cloning via Coqui XTTS from a mono 16-bit WAV sample.
• Customizable AI personality through chat_params.json.
• Requires roughly 8 GB of VRAM for real-time response.
• Non-commercial only under the Coqui Public Model License.

Caveats

• Explicitly labeled “experimental alpha” with known XTTS glitches and answer quality the author admits falls short of GPT-4 or Claude.
• A known incompatibility between newer transformers releases and Coqui TTS can break synthesis; the README suggests pinning an older version or upgrading RealtimeTTS to resolve it.
• Restricted to non-commercial use by the Coqui license.

Verdict

Tinkerers with a decent GPU and a tolerance for alpha software will get the most out of this. If you need polished voice AI or a commercial license, look elsewhere.