The glue between your ESP32 and a dozen Chinese AI APIs

What it does

A backend companion for the xiaozhi-esp32 open-source voice-assistant hardware. It handles the unglamorous wiring between ESP32 devices and cloud AI services: streaming speech recognition, LLM dialogue, text-to-speech, and vision processing. A web console lets you manage users, devices, and agents, while an MQTT gateway pushes commands back to hardware.

The interesting bit

Voiceprint recognition runs in parallel with ASR, so the LLM knows who is talking before it replies. The project exposes multiple MCP (Model Context Protocol) interfaces—client, server, and access-point—so the ESP32 can trigger external tools and knowledge bases. Hobbyists can stick to an all-free tier using local FunASR and EdgeTTS, or switch to streaming Chinese cloud APIs for lower latency.

Key highlights

• Streaming ASR/TTS pipeline that cuts response latency by roughly 2.5 seconds compared to earlier releases
• Multi-user voiceprint enrollment and recognition, processed in parallel with speech-to-text
• Pluggable backends for Chinese LLMs and speech services (Zhipu, Alibaba Qwen, Xunfei, ByteDance Huoshan)
• Built-in RAGFlow knowledge-base integration and optional mem0ai or PowerMem memory modules
• Web console with i18n support and an MQTT+UDP gateway for device command dispatch

Caveats

• The authors explicitly warn the software is unfinished and has not passed network-security assessment; they advise against production or public-internet use without hardening.
• It is built specifically for the xiaozhi-esp32 firmware ecosystem, so without the hardware this is not a generic voice-assistant framework.

Verdict

Worth a look if you already own the hardware and want to self-host your own backend. Skip it if you need production-grade security today or are searching for a general-purpose voice platform.