An Async Python Assistant That Speaks to Desktops and Robots

What it does

Built on Python’s asyncio stack, py-xiaozhi handles real-time voice streaming with Opus compression, camera-based vision-language tasks, and IoT device control. It deploys as a PySide6 desktop GUI, a headless CLI, or a raw GPIO interface for embedded boards such as the Raspberry Pi and Jetson Nano, effectively wrapping an LLM pipeline in whatever skin the hardware demands.

The interesting bit

The project evolved from the xiaozhi-esp32 firmware and is now the upstream dependency for D-Robotics’ RDK stack. It treats voice as a control plane for hardware: offline wake-word detection via Sherpa-ONNX triggers an async pipeline that can return synthesized speech, invoke MCP tools over JSON-RPC, or toggle physical pins on a robot. Keeping that many subsystems from blocking each other is the real trick, handled here with dependency injection and an event-driven plugin architecture.

Key highlights

• Sub-20 ms voice latency with 5 ms audio processing queues and Opus codec streaming.
• Three UI modes—PySide6/QML GUI, CLI, and embedded GPIO—running off the same core.
• Bundled MCP tool server covering camera, screenshot, volume, weather, and app management.
• Offline wake-word detection via Sherpa-ONNX plus encrypted WebSocket/MQTT transport.
• Adopted by D-Robotics as the upstream base for their embedded voice-assistant work.

Caveats

• The README warns that main-branch updates require manual dependency reinstallation, so staying current is not hands-off.
• A stable internet connection is listed as a basic requirement for AI services; fully offline LLM inference is not advertised.
• Python support is explicitly capped at versions 3.10 through 3.12.

Verdict

This is for developers prototyping voice-driven robotics or edge AI who want the full stack—wake word, audio codec, protocol transport, and hardware I/O—in one Python repo. If your assistant lives only in a browser tab, look elsewhere.