A Unity VTuber who actually listens (and talks back)

What it does

This Unity project wires a Live2D avatar to Azure speech-to-text, OpenAI’s chat API, and Azure text-to-speech. The avatar lip-syncs in real time, presumably driven by audio or facial analysis, while you hold a conversation with a language model. It’s essentially a fully voiced chatbot with a face—think desktop companion or streaming sidekick.

The interesting bit

The project grafts together three separate cloud APIs and a real-time rendering pipeline, which is more plumbing than magic, but the integration is where the work lives. The authors also documented their development process in a technical writeup, which is rarer than it should be for student projects.

Key highlights

• Live2D model with real-time blendshape animation (lip sync, expressions)
• Speech pipeline: Azure STT → OpenAI GPT → Azure TTS
• Includes a settings UI that persists API keys locally
• Ships with a specific “桃花酪元子” character model and custom background art
• Academic project from Nanjing University, MIT-licensed code

Caveats

• Requires three separate API keys (Azure, OpenAI, APISpace) and manual configuration; no shared keys provided
• Depends on a third-party OpenAI proxy for mainland China access, which may be brittle
• Unity 2021.3.0 and a paid Asset Store package (OpenCVPlusUnity) are hard dependencies

Verdict

Worth a look if you’re building a VTuber rig, a kiosk avatar, or just want to see how Live2D + LLM + cloud speech services bolt together. Skip it if you need something turnkey or are allergic to API key juggling.