16 tokens of memory instead of 128K in your context window

What it does Memorilla is a trained memory module that sits in front of a frozen LLM (Qwen3-8B by default). It reads a whole document collection — a user’s chat history, a book, an agent’s episode log — and distills it into 16 query-dependent memory tokens prepended to the prompt. The LLM never changes; only the memory module is trained, in three stages over Wikipedia, a mixture, and multitask data.

The interesting bit The compression is fixed-size: 16 tokens whether you have 10 documents or a million, so latency stays flat as the collection grows. On FactKG it actually beats stuffing every document into the prompt (84.33% vs. 80.43%) with 108× fewer tokens, and on NarrativeQA it compresses books by more than 4,000×. It also trains from reward alone via GRPO on TextWorld, lifting agent success from 18.6% to 24.2%.

Key highlights

• Best average performance vs. Mem0 and MemoRAG at a 16-token budget
• Response time stays at 0.74s from 1 to 512 documents (~128K tokens); full context grows to 19.9s
• Connects indirect clues — e.g. infers a family history of diabetes from an old unrelated question
• Ships with PersonalizationV4, a new 149-user synthetic personalization benchmark
• Checkpoints on Hugging Face; custom datasets work via a parquet layout with no code changes

Caveats

• Training is heavy: ~19 hours on 4× H100 80GB and ~310 GiB of downloads; evaluation alone needs an 80GB GPU
• Linux + CUDA + Python 3.11 only; FlashAttention-2 is the default for training (PyTorch SDPA works as a fallback)
• The RL side depends on a separate fork of SkyRL, so that path is more plumbing than the rest of the repo

Verdict Worth a look if you’re building personalized assistants or long-history agents and are tired of paying for full-context prompts. Skip it if you wanted a drop-in RAG replacement that runs on a laptop.