A Memory Tree for Agents That Trims Token Burn by Seventy-One Percent

The Hype Moment and the Hard Receipts

Open-sourced in late March 2026 by the HITsz-TMG research group, LycheeMemory arrived with an unusually specific set of receipts. On PinchBench—a benchmark covering more than five hundred models for OpenClaw agent task execution—the framework lifted task scores from 43.22 percent to 48.98 percent, a six-point gain that arrived alongside a seventy-one percent collapse in token consumption and a fifty-five percent drop in cost. The team translates this into a 5,480 percent ROI: one yuan sunk into embeddings supposedly saves nearly fifty-five in LLM inference spend. Those numbers explain the sudden attention spike. But the more interesting question is what architectural choices produced them.

LycheeMemory is not an isolated hack. It belongs to the third-generation Lychee large-model series, a family of projects that includes LycheeDecode for KV-cache selective recall and LycheeCluster for hierarchical context indexing. The rebrand from LycheeMem to LycheeMemory in April 2026 signaled a shift from experimental repository to packaged infrastructure: a PyPI distribution, an MCP server, native plugins for OpenClaw and Claude Code, and a web demo. The project wants to be treated as memory infrastructure, not a sidecar script.

Beyond the Goldfish Bowl

The problem LycheeMemory targets is now familiar across the agent ecosystem. Large language models are stateless. Even with context windows stretching to 128,000 or 200,000 tokens, models suffer from a lost-in-the-middle effect, with reliability typically degrading well before the advertised limit. Industry commentary has labeled the broader syndrome the goldfish problem: technically capable agents that forget duplicate customer complaints across conversations because they possess no durable, structured memory of their own. LycheeMemory’s response is to treat memory not as a database table bolted onto an agent, but as a tiered cognitive architecture.

The system divides memory into four stores. Working memory holds the active session under a dual-threshold token budget: at seventy percent usage it quietly pre-compresses older turns, and at ninety percent it blocks the pipeline to flush history into summary anchors. Semantic memory is the core long-term store, built from seven typed record categories—facts, preferences, events, constraints, procedures, failure patterns, and tool affordances—each carrying action-facing metadata and usage statistics. Procedural memory caches reusable skills as structured entries with HyDE retrieval. A newer visual memory layer stores VLM-generated captions and CLIP embeddings for images, complete with an Ebbinghaus-inspired forgetting curve that soft-expires low-importance entries after roughly ninety days. The storage substrate is deliberately lightweight: SQLite with FTS5 for text, LanceDB for vectors, and the local filesystem for raw images. No graph server is required.

Typed Records and the Memory Tree

Where LycheeMemory diverges from the typical approach of dumping embeddings into a vector database is in its insistence on typing and structure. Every piece of knowledge ingested from conversation passes through a single-pass encoder that expands pronouns into full expressions, assigns a semantic category, and annotates action metadata such as tool tags and constraint flags. Each record is hashed by the SHA-256 of its normalized text, making ingestion naturally idempotent.

The genuinely unusual mechanism is the Record Fusion Engine, which runs after consolidation without calling an LLM. It searches for near-duplicates with cosine similarity above 0.85 and soft-expires them, then clusters surviving records above 0.75 similarity into connected components. Each cluster becomes a CompositeRecord—a denser summary that preserves child pointers to its atomic members. The engine then runs the same clustering pass over the composites themselves, building a hierarchical memory tree that abstracts upward. Retrieval operates first at the composite level: an approximate-nearest-neighbor pre-filter narrows the field to twenty candidates, a single LLM call judges holistic relevance and flags entries too abstract to stand alone, and the engine expands downward through the tree to surface granular records only when the query demands detail. A reflection loop supplements any remaining gaps with full-text and vector search over raw episode turns.

This hierarchy is the project’s central bet. Flat vector stores treat memory as a bag of summaries; LycheeMemory treats it as an index that can be navigated top-down, trading the cost of embedding math and one LLM relevance judgement for the far higher expense of stuffing every possibly-relevant turn into the context window.

A Pipeline, Not a Library

LycheeMemory is not merely a storage layer. It ships as a complete five-stage pipeline. A rule-based working-memory manager handles token budgets. A search coordinator plans retrieval conditioned on an action state—recent failures, tool use, constraints—and executes the hierarchical semantic search plus HyDE-driven skill retrieval. A synthesizer agent acts as an LLM-as-Judge, scoring each retrieved fragment on an absolute zero-to-one relevance scale and fusing the survivors into a dense background context string with full provenance. Only then does a reasoning agent generate the final response. After the reply is sent, a consolidator agent runs asynchronously in a thread pool, performing novelty checks, compact consolidation, and skill extraction without blocking the user-facing latency path.

The synthesizer stage is where the token savings likely materialize. By judging relevance before injection, the pipeline avoids paying for tokens that look semantically similar but carry no actionable information. The background consolidation stage, meanwhile, uses the search-stage raw semantic fragments for its novelty check rather than the answer-time synthesized context, preventing the model from discarding valid new information simply because it was not needed for the immediate reply. The OpenClaw plugin goes further: it mirrors turns automatically and triggers boundary consolidation when a session ends, so the agent does not have to remember to remember.

Plugs and Protocols

LycheeMemory’s packaging strategy reveals its intended role. It runs as a local HTTP service exposing an OpenAPI interface, an MCP server, and native plugins for OpenClaw, Claude Code, and Hermes. Any MCP client can discover its tools and treat long-term memory as a standardized resource rather than a bespoke integration. This places it in a small but growing cohort of projects—Pixeltable is another—that use MCP to expose persistent memory and retrieval to agents.

The timing is deliberate. The agent ecosystem is shifting from isolated models toward protocol-driven composition. Anthropic’s Model Context Protocol has become a default lingua franca for exposing structured context, while Google’s Agent-to-Agent protocol targets cross-platform interoperability. LycheeMemory rides this wave, though it also inherits the risks. Security analyses have flagged MCP’s expanded attack surface: prompt injection can now trigger real actions or leak sensitive data from live systems. LycheeMemory’s documentation emphasizes scope limitation and session isolation, but the project spends more time on feature lists than on threat modeling.

Rough Edges and Open Questions

For all its architectural ambition, the project is clearly early. The transformer reranker is labeled experimental and v0, with a Hugging Face checkpoint that loads on first use if PyTorch is present and falls back silently if not. Visual memory offers a fast mode that compresses images to 512 pixels at JPEG quality 75 and caps VLM timeouts at fifteen seconds—usable, but hardly production-grade for high-fidelity multimodal workflows. The hierarchy depth and clustering thresholds are fixed constants that may not transfer across domains with different semantic densities. The system also depends on litellm for provider routing, inheriting the compatibility matrix and failure modes of that abstraction layer. And while the PinchBench numbers are impressive, they derive from a single benchmark tied to one agent runtime; broader replication across independent MCP clients remains to be demonstrated.

Outlook: From Storage to Learning

LycheeMemory’s most forward-looking feature may be the telemetry it collects without yet fully exploiting. Every semantic record carries usage statistics—retrieval counts, action success counts, failure tags—and retrieval logs persist the action state, plan, and user feedback. The authors describe these as RL-ready statistics, implying a future phase where the memory system does not merely retrieve, but learns what to keep, what to compress, and when to forget. That would move the framework from a passive retrieval layer into an active component of agent optimization, aligning with the broader Lychee research program into memory intelligence and continual learning.

For now, it occupies a crowded but strategically vital niche. Between flat vector databases, in-memory chat histories, and sprawling knowledge graphs, LycheeMemory has planted a flag for structured, hierarchical, action-aware memory. Whether the fifty-four-to-one ROI holds outside PinchBench is an open question. The architecture, at least, is a serious attempt to answer it.