Clone a voice into six languages the model never studied

What it does OpenVoice is a text-to-speech system that clones a speaker’s tone color from a short audio sample. It separates voice timbre from style parameters—emotion, accent, rhythm, pauses, and intonation—so you can synthesize speech in the cloned voice while adjusting those traits independently. V2 adds native support for English, Spanish, French, Chinese, Japanese, and Korean and re-licensed the whole project under MIT for commercial use.

The interesting bit The zero-shot cross-lingual claim is the standout: neither the reference audio nor the target language needs to have appeared in the massive multi-speaker training set. It is essentially doing voice transfer with a passport full of visas it never applied for.

Key highlights

• Accurate tone-color cloning from a reference speaker
• Granular control over emotion, accent, rhythm, pauses, and intonation
• Zero-shot cross-lingual synthesis where target and reference languages can be absent from training data
• V2 improves audio quality and natively supports six languages
• MIT licensed for commercial use as of April 2024

Verdict Worth a look for developers who need instant, multilingual voice cloning with commercial-friendly licensing. Less interesting if you are hunting for a ground-up audio model to fork as a research baseline.