snap-stanford/memorilla · 18 Nov 2026 · Feature

Sixteen Tokens Against the Context Window

Stanford's Memorilla compresses an entire document collection into a fixed handful of learned memory tokens for a frozen LLM — and sometimes beats reading everything.

snap-stanford/memorilla
★1.6k stars Velocity · 7d +115 ★/day
star history

The context window has become the load-bearing wall of modern AI, and everyone keeps trying to knock it out. The latest attempt comes from Jure Leskovec’s group at Stanford: Memorilla, a memory system that reads an entire document collection — a user’s chat history, a novel, a movie script — and hands a frozen LLM exactly sixteen tokens to work with. Not sixteen thousand. Sixteen.

snap-stanford/memorilla

The repo, snap-stanford/memorilla, landed alongside the paper in late 2026 with trained checkpoints, a benchmark dataset, and a reinforcement-learning pipeline. It is a research artifact with unusually good manners: Apache-2.0 code, permissively licensed data, and documentation that explains what the thing actually does. But the reason people are paying attention is a stranger claim than “smaller is faster”: on at least one benchmark, sixteen learned tokens outperform giving the model every document.

The problem it refuses to solve the usual way

A frozen LLM can only use what fits in its context. The industry’s standard answers are to make the window bigger, to retrieve a few relevant chunks (RAG), or to bolt on a memory layer that summarizes past interactions. All three work, with familiar trade-offs. Long windows are slow and expensive — attention cost grows quadratically with input length, as IBM’s Rogerio Feris puts it. RAG is fast but brittle: it shines when the answer is verbatim in a document and stumbles when the question requires connecting weakly related facts scattered across a collection.

Memorilla’s authors start from four design requirements that, taken together, rule out most existing approaches: every query should be able to see the whole collection, recall should depend on the query, the token budget should stay fixed no matter how large the collection grows, and document embeddings should be computed once and reused. That last requirement is the quietly important one. Adding a document costs a single encoder pass; the store grows without re-encoding the past, which is what makes the scheme plausibly scale to millions of tokens.

The architecture is a division of labor. A frozen encoder (Qwen3-Embedding-4B in the released checkpoints) stores each document as an embedding. When a question arrives, a trained memory module reads the question and the whole embedding store, and produces sixteen memory tokens that get placed directly into the LLM’s input embedding space, alongside the query. The LLM itself — a frozen Qwen3-8B — never changes. Only the memory module is trained, in stages that move from paraphrasing Wikipedia to general QA and summarization to benchmark training splits. The training recipes are staged deliberately: a curriculum, not a single loss.

The interesting part is what those sixteen tokens are. They are not text. They are not retrieved passages. They are latent vectors — a learned, query-dependent distillation of an entire collection, positioned in the embedding space the decoder already understands. In effect, Memorilla trains a small network to speak fluent embedding at the LLM.

The result that raises eyebrows

The headline numbers from the paper: with sixteen memory tokens, Memorilla achieves the best average performance against methods including Mem0 and MemoRAG, winning five of eight benchmarks spanning personalization, long-document QA, and knowledge-intensive QA. On FactKG it reaches 84.33% accuracy versus 80.43% for full-context inference — while using 108× fewer tokens to represent the source documents.

Beating full context with a 108× compression is the kind of claim that invites skepticism, and it should. The charitable reading is not that sixteen tokens carry more information than 67,000. It’s that full context is not actually a clean upper bound. Long contexts dilute attention, and models degrade when the needle sits in a large haystack. A learned compression that has been trained to surface what questions typically need can, in aggregate, beat a model drowning in raw text. On NarrativeQA, Memorilla compresses books and scripts by more than 4,000× — sixteen tokens against 67,385 — “without sacrificing performance,” per the project site.

The latency story is more straightforwardly good. On one H100, growing a collection from 1 to 512 documents (about 128K tokens) leaves Memorilla’s response time at 0.74 seconds, while full context slows from 0.75 to 19.9 seconds — 27× slower. Across eight benchmarks, the geometric-mean token usage is 113× below a RAG baseline retrieving fifty documents. Whether that matters for your deployment depends on whether your bottleneck is prefill compute, and for long-context personalization workloads, it usually is.

The personalization demo, and why it lands

The README’s showcase example is worth dwelling on because it illustrates exactly the failure mode of retrieval. A user with 107 past conversations — roughly 32,000 tokens — asks for breakfast ideas that will keep them full. The user never mentions their family history of type 2 diabetes. The only clue is buried in one old conversation about whether maternal versus paternal diabetes history raises risk more. Memorilla connects that thread to an unrelated breakfast question and suggests meals that balance protein, fat, and fiber to avoid blood-sugar spikes. The unmodified Qwen3-8B answers with a generic breakfast list.

This is the connective-tissue problem that the memory-versus-retrieval literature keeps circling: retrieval is an open-book test, memory is knowing the student. A RAG system retrieving top-k chunks for “breakfast ideas” would almost certainly not surface a months-old question about diabetes heritability — the lexical and semantic distance is too large. Memorilla’s query-dependent compression reads the whole store every time, so weakly related facts stay in reach.

To probe this capability, the team released PersonalizationV4, a synthetic benchmark of 149 fictional users, each with 200 prior conversations. Questions are written to require combining at least two facts about the user that the conversations show but never state. It is a benchmark designed to fail retrieval, which is either rigorous methodology or stacking the deck, depending on your allegiances — probably both.

A crowded field, a different bet

Memory augmentation for LLMs is not new territory, and Memorilla arrives into a field with strong priors. LongMem (NeurIPS 2023, Microsoft and UCSB) froze the backbone as a memory encoder and trained a residual side-network as retriever and reader, caching long-term context without staleness. IBM’s CAMELoT plugs an associative memory module into a pretrained Llama 2-7B with neuroscience-inspired consolidation, novelty, and recency rules, cutting perplexity by up to 30%; its sibling Larimar optimizes for fast fact updates and forgetting. A 2025 memory-augmented architecture from Newton AI Lab and KAIST attacks memory bloat with relevance-based pruning rather than LRU eviction. And the applied world, per practitioner discussions, has largely converged on vector stores plus summarization plus forget-mechanisms, assembled in LangChain or LlamaIndex.

Most of these systems share an assumption: memory should be inspectable, editable text — summaries, facts, slots. Memorilla’s bet is the opposite. Its memory is latent, opaque, and query-dependent. You cannot open the drawer and read what it remembered, because what it remembered doesn’t exist until a question arrives. That buys the fixed budget and the collection-wide access, at the cost of the auditability that production personalization systems often want — and increasingly need, given privacy regulation around user profiling. A 2025 framework paper from Mercedes-Benz and Ulm University on personalized agents emphasizes self-validation and evolving user profiles; Memorilla offers none of that machinery, just a very good compressor.

The authors are candid about where the fixed budget hurts. Their own comparison concedes that RAG suits questions with verbatim answers, while Memorilla’s sixteen tokens can lose fine details such as exact names. Sixteen vectors is a hard ceiling; a question demanding the third name in a list of forty may simply not fit. RAG and Memorilla can share the same embeddings and the same frozen LLM, which suggests the honest deployment is both — retrieval for the verbatim, latent memory for the connective — though the paper’s timing comparisons notably left RAG untimed.

The agent angle

The most speculative branch is reinforcement learning. Memorilla can serve as an agent’s memory and learn from reward alone: the repo trains the memory module with GRPO on TextWorld text adventures, via a fork of SkyRL. The result is a modest but real shift — average success rate rises from 18.6% with full history in context to 24.2% with eight trained memory tokens. The mechanism is intuitive: an agent that can compress what it has already explored stops re-walking cleared rooms. The README’s replay shows exactly this — the full-history agent loops through visited rooms until the turn limit; the memory-augmented one heads for the goal.

The numbers are small, and TextWorld is a toy, but the shape of the result matters. Learning a memory policy from environment reward, rather than from supervised QA pairs, is the difference between a memory that answers questions well and a memory that acts well. If latent memory turns out to be trainable by RL at scale, the implications run well past chatbots.

Rough edges and open questions

This is a research release, and it wears that honestly. Training from scratch assumes Linux, CUDA, Python 3.11, and about 310 GiB of downloaded data and embeddings; the recipes were run on four H100 80GB GPUs over roughly nineteen hours. Evaluation is lighter — fifteen minutes per checkpoint on one GPU — but the hardware floor is real. The single-task recipe warm-starts from a Stage 2 checkpoint, so “using your own data” means fine-tuning on top of their curriculum, not training from zero, unless you want to pass an empty init flag and find out why they don’t recommend it.

The deeper open questions are architectural. The memory module is trained against one decoder family; how well sixteen learned tokens transfer to a different frozen LLM is unaddressed in the provided material. The opacity problem — no human-readable memory to inspect, correct, or delete — is a genuine product liability in any regulated personalization setting, and the sources don’t show a workaround. And the FactKG win over full context, while striking, is one benchmark; the honest summary is that Memorilla wins five of eight and trades blows elsewhere.

What Memorilla establishes is a proof of concept with unusually sharp edges: that a fixed, tiny, learned, query-dependent latent budget can stand in for arbitrarily large context — and occasionally beat it. The context window will keep growing, and vendors will keep selling bigger ones. The interesting question Memorilla poses is whether size is the wrong axis entirely — whether the right unit of memory was never tokens of text, but directions in embedding space, chosen on demand. Sixteen of them, it turns out, go a long way.

Sources

  1. [2610.21375] Memorilla: Latent Semantic Memory for LLMs
  2. Augmenting Language Models with Long-Term Memory
  3. Memory vs Retrieval Augmented Generation
  4. Augmenting Language Models with Long-Term Memory
  5. - Memory Implementation in LLM-based RAG: A Quick Guide: | Anshul Yadav
  6. Memorilla: Latent Semantic Memory for LLMs : r/singularity
  7. Memory-Augmented Architecture for Long-Term Context Handling in ...
  8. How are you handling "memory" and personalization in your end-user AI apps?
  9. Stanford Research Open-Sources Memorilla, a Latent Memory Module ...
  10. How memory augmentation can improve large language models
  11. Enabling Personalized Long-term Interactions in LLM-based Agents ...
  12. Latent Semantic Memory for LLMs: MEMORILLA

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.