Speech synthesis that laughs, pauses, and switches speakers

What it does

ChatTTS is a text-to-speech system built for dialogue rather than audiobooks. It targets LLM assistant scenarios, generating speech in English or Chinese with support for multiple speakers and conversational prosody. The repository holds the core inference code and pre-trained models; polished end-user products live in a separate community index.

The interesting bit

Instead of flat narration, the model predicts fine-grained prosody—laughter, pauses, and interjections—through special tokens like [laugh] and [uv_break]. The authors claim it surpasses most open-source TTS models on prosody, though they concede English support is still experimental.

Key highlights

• Trained on 100,000+ hours of Chinese and English audio; the open-source release is a 40,000-hour base model without supervised fine-tuning.
• Token-level control via markers for laughter, breaks, and oral style to shape delivery.
• Needs at least 4GB VRAM for 30-second clips; runs at roughly 0.3 real-time factor on an RTX 4090.
• Code is AGPLv3+, but model weights are CC BY-NC 4.0—strictly academic and non-commercial.
• The authors intentionally degraded the open weights with added noise and heavy MP3 compression to deter deepfake abuse.

Caveats

• Autoregressive instability is acknowledged: multi-speaker output and audio quality can vary, so plan to sample multiple times.
• English is labeled experimental, and the current release only offers limited emotional controls (laugh, break).
• Optional optimizations like FlashAttention-2 and TransformerEngine are explicitly marked broken or slower; the README warns not to install them.

Verdict

Experiment with it if you’re building non-commercial voice UIs for Chinese or English conversational agents. Look elsewhere if you need stable, commercial-grade TTS or primarily English content.