Pocket-sized anime companion assembled from spare AI parts

What it does

ChatWaifu_Mobile is an Android app that pipes ChatGPT responses through locally synthesized anime voices and native Live2D avatars. You speak to it via on-device Sherpa-ncnn recognition, it thinks via the OpenAI API, and it answers back as one of several built-in characters—complete with lip-sync that the author admits is basically faked. The UI is Jetpack Compose over Retrofit, MVVM, and Room, while the heavy native inference engines run in separate processes glued together by AIDL.

The interesting bit

The project is less a chatbot and more an integration stress test: it binds VITS-ncnn, Live2D, Sherpa-ncnn, and Meta’s lipSync into a single Android process group, then papers over the seams with Baidu translation so Japanese voice models can pronounce whatever ChatGPT outputs. That it keeps voice and graphics fully offline while still needing an OpenAI key for the brain creates a curious split-brain architecture.

Key highlights

• Local VITS-ncnn speech synthesis and Sherpa-ncnn voice recognition; only the language model hits the cloud
• Native Live2D rendering with swappable character models in standard .moc3 and .model3.json formats
• Built-in characters from Blue Archive, Steins;Gate, and ATRI, plus support for custom VITS and Live2D imports
• AIDL-based IPC because multiple ncnn libraries appear to hog process-exclusive resources
• Optional Baidu Translate layer to force ChatGPT output into Japanese for the built-in voice models

Caveats

• The lip-sync integration is acknowledged as broken: timing mapping was “too troublesome,” so the avatar just plays a looping animation
• Built-in VITS models are Japanese-only; the README warns that other languages sound “quite一言难尽” and relies on Baidu translation to bridge the gap
• Replacing models requires manual file-system placement and strict name matching between the VITS directory and Live2D model3.json

Verdict

Worth a look if you are building Android-native AI companions and need a reference for taming multiple ncnn runtimes. Skip it if you want production polish or a voice assistant that speaks your language out of the box.