An open-source AI VTuber that sings, draws, and reads your danmaku

What it does

AI-YinMei (“Yinmei”) is a Python project that turns an LLM into a livestreaming anime character — a VTuber, desktop pet, and chatbot in one. It aggregates live chat (danmaku) from Bilibili, Douyin, Kuaishou, Douyu, Huya, WeChat live, and QQ bots, then responds with streaming voice, Live2D expressions, songs, drawings, and video clips. It also works as a plain desktop pet with its own microphone pipeline.

The interesting bit

The engineering claim to fame is latency: fully streaming voice dialogue at a claimed ~600ms response, a 10ms multi-head-attention intent classifier that does intent and emotion classification together, and tool selection in 200–500ms even with 100 tools mounted. The author also ships a modified cosyvoice2 TTS model, fine-tuned for emotional delivery and accelerated with vLLM.

Key highlights

• Aggregates chat from 9 sources into one stream, with a points economy (chatting earns, singing costs)
• ~27 LLM tools including MCP support, camera vision, browser control, and even keyboard/mouse control of the host PC
• Long-term memory with selective recall by time expression, plus full chat persistence in MongoDB
• “Diffusion thinking” builds word-relation graphs in Neo4j, trainable from QQ group data
• Song learning in ~150–200 seconds per track, with trainable voice timbres

Caveats

• The GitHub open-source version is 1.8.1 and lacks the admin backend; the full 3.0 version (intent analysis, voiceprint, points system, etc.) is closed-source and distributed via Baidu/Quark netdisk links
• Documentation, videos, and community are almost entirely in Chinese
• Some features lean on external heavyweight stacks: FastGPT for the knowledge base, Neo4j, Stable Diffusion, GPT-SoVITS — this is an orchestration layer as much as a product

Verdict

Worth a look if you build AI streamers or want a very complete reference for low-latency voice-agent plumbing. If you want a fully open, English-documented project, the closed-source full version and language barrier will test your patience.