Puppeteer Unreal avatars with a webcam and offline AI

What it does MediaPipe4U is an Unreal Engine plugin that turns a standard RGB webcam into a real-time motion-capture and facial-tracking rig. It drives 3D avatars with body, finger, and face data—including ARKit blendshapes—and adds offline text-to-speech, speech recognition with wake-word support, and Ollama-powered LLM dialogue entirely inside the editor. Everything is packaged as pure C++ plugins, so you do not need Python or external programs running alongside your project.

The interesting bit The plugin’s real trick is cramming a full offline AI media pipeline—vision, voice, and language—into a game engine, letting you puppeteer avatars and hold spoken conversations without cloud dependencies. It even ships with an open-source Android remoting app so you can use a phone as a wireless face-capture camera.

Key highlights

• Real-time body and face capture from webcams, video files, or RTMP streams, with ControlRig and Live Link support.
• Offline TTS and ASR with multiple model backends (including Kokoro, Melo, FunASR, and SenseVoice) and a voice wake-up feature.
• Integrated Ollama support for local LLM inference and a Blueprint dialogue component that wires LLM, TTS, and ASR together.
• Pure C++ implementation for Unreal Engine 5.1 through 5.6; no Python or external processes required.
• Supports Windows and Android (with an open-source remoting app), though Linux support is marked coming soon.

Caveats

• Google’s Holistic Task API currently lacks GPU inference, so Android falls back to CPU-only processing.
• On UE 5.4 and newer, the editor may appear to hang at 75% on first launch while it downloads and compiles OpenCV source code locally.
• Documentation is primarily in Chinese; the English version is auto-translated.

Verdict Unreal developers building interactive avatars, virtual production, or AI-driven NPCs should look here—especially if they need everything offline. If you are not working in Unreal Engine, or only need a single-purpose tool like a standalone mocap app, this is not for you.