From JPEG to lip-sync video, wired for production

What it does

AvatarAI turns a static face photo and a few seconds of audio into a real-time conversational avatar. The pipeline runs speech through Whisper, generates a response via Claude, GPT-4o, or a local Ollama model, clones voice with Chatterbox, and syncs lips with MuseTalk V1.5 — all streamed over WebSocket so the face starts moving before the LLM finishes its reply. The README claims a sub-four-second time-to-first-video-chunk on an AWS GPU instance.

The interesting bit

Most open-source talking-head projects stop at a Gradio demo; this one adds JWT cookies, Postgres migrations, Prometheus metrics, Terraform for AWS GPU deploys, and a pytest suite. The sentence-level streaming architecture is the clever part: the backend splits LLM output into chunks, runs TTS and lip-sync per sentence, and pushes video chunks to the browser while the model is still streaming tokens. It also handles barge-in — talking over the avatar cancels the in-flight turn.

Key highlights

• Runs fully offline via USE_LOCAL_STORAGE=true with local Whisper and Ollama, or scales to AWS g5.xlarge with CUDA 11.8
• Zero-shot voice cloning from 10–60 seconds of audio in 23 languages using Chatterbox Multilingual
• TTS fallback chain — if Chatterbox fails, it drops through edge-tts and gTTS so the avatar never goes silent
• Production scaffolding included: per-user rate limiting, Alembic migrations, S3/CloudFront assets, Celery queues, and nginx reverse-proxying WebSockets
• Persistent MuseTalk worker keeps models loaded in memory rather than paying the startup tax on every utterance

Caveats

• The MuseTalk models alone weigh roughly 9 GB, and the full system needs Docker Compose, PostgreSQL, Redis, and optionally CUDA
• Chatterbox TTS runs in a separate Python virtual environment, suggesting dependency isolation headaches
• Performance claims like sub-four-second latency and 30 FPS are explicitly tied to AWS GPU instances; the README does not quantify CPU fallback speed

Verdict

Teams who need a branded digital-human kiosk or customer-service avatar and want to own the stack rather than rent a SaaS should look here. Tinkerers after a lightweight weekend project should look elsewhere — this is a full distributed system in a box.