A split-brain memory bank for voice agents that never hits pause

What it does VoiceMem is a long-term memory layer purpose-built for real-time voice agents. It ingests audio or text, extracts facts, speaker identity, emotion, and sound events, and stores them in a structured graph. When the agent needs context, it retrieves only the top few relevant memories and injects them into the conversation. The entire pipeline is streaming, so retrieval can happen while the user is still talking.

The interesting bit Instead of flattening memories into one vector pile, VoiceMem splits them into a “left brain” for factual schema-and-entity graphs and a “right brain” for emotional and relational context. The README reports that this dual-hemisphere design, plus speculative prefetching, keeps median retrieval latency around 12 ms and per-query token usage near 430—both substantially leaner than the Mem0 figures it quotes.

Key highlights

• Dual-hemisphere storage: a “left brain” for factual entities and schemas, and a “right brain” for emotions, personality, and cross-entity relationships.
• Streaming-first pipeline that extracts and indexes memories from live audio while the user is still speaking.
• Reported benchmark results: 91.2% on LoCoMo and 69.44% on PersonaMem, using only the top five retrieved memories.
• Ships with fine-tuned Qwen-Omni and Step-Audio2-Mini adapters, plus a 400K-sample ChatMem-400K dataset for training your own.
• Memory retrieval runs fully offline; only the fact-extraction write path requires an external LLM API.

Caveats

• Still on v0.0.2, so expect the schema and APIs to evolve quickly.
• Release notes in the README are dated 2026, which suggests either a typo or an impressively long roadmap.
• End-to-end offline use is not yet possible out of the box: memory retrieval runs locally, but the write path currently needs an external LLM for fact extraction.

Verdict Give VoiceMem a spin if you are building a real-time voice agent that needs to remember facts, feelings, and soundscapes without bloating the context window. If your stack is text-only or stateless, it is probably overkill.