Fay: Open-Source Plumbing for Digital Human Agents

What it does Fay is middleware for digital humans. It sits between a 2.5D or 3D avatar frontend and the messy backend of LLMs, speech recognition, text-to-speech, and business APIs. The framework exposes unified interfaces—text, voice, avatar driving, management, and broadcast—so the same agent can run on a microcontroller, a mobile app, or a website without rewriting the orchestration layer.

The interesting bit The project treats the digital human as an I/O and scheduling problem rather than a graphics problem. It decouples the avatar renderer from the brain, letting you hot-swap models like DeepSeek or OpenAI-compatible LLMs, swap ASR/TTS engines, or run the entire stack offline. It also layers in agentic behavior—MCP tool management, biomimetic memory, scheduled proactive dialogue, and wake-word interruption—which feels closer to a voice operating system than a chat wrapper.

Key highlights

• Fully offline capable; no mandatory cloud dependencies.
• Pluggable model slots for LLMs (OpenAI-compatible, DeepSeek), ASR, and TTS.
• Multi-user, multi-concurrency support with streaming responses.
• Agent tool-calling with MCP (SSE/Studio) and a built-in config management center.
• Broad terminal coverage: microcontrollers, mobile, web, big screens, and third-party systems.

Caveats

• Documentation and community resources live on Feishu (Lark), which may be cumbersome for developers outside China.
• The README warns that public resource keys are “very slow,” so expect to bring your own API keys and models.
• Much of the value depends on integrating external digital human renderers and speech models that are not bundled in the repository.

Verdict Fay is worth a look if you are building a kiosk, virtual anchor, or customer-service avatar and need an open, self-hosted orchestration layer. If you just want a drop-in chatbot widget, this is over-engineered.