LLMs trading thoughts in KV-cache, skipping English entirely

What it does

Cache-to-Cache (C2C) treats the KV-cache as a native inter-model protocol. Instead of having one LLM generate text that another LLM tokenizes and re-encodes, C2C projects the sharer’s KV-cache directly into the receiver’s semantic space and fuses them. A lightweight RosettaModel wrapper handles the translation, and only the projector networks are trained—both source and target LLMs stay frozen.

The interesting bit

The projectors act like a Rosetta Stone for latent spaces: they learn to align different architectures’ internal representations so a Qwen2.5 model can literally whisper its partial thoughts into a Qwen3 model’s attention layers. The README claims this yields 8.5–10.5% higher accuracy than either model alone and roughly 2× lower latency than text-based hand-off.

Key highlights

• Pre-trained fusers exist on HuggingFace for specific Qwen3 receiver pairings (Qwen2.5, Llama-3.2, Qwen2.5-Math, and larger Qwen3 variants).
• Supports multi-sharer fusion—combining KV-caches from several teacher models into one receiver—though the README flags this as preliminary.
• Training is projector-only; you don’t need to fine-tune the base LLMs.
• A live Gradio demo and interactive chat script are available for side-by-side comparison.
• The authors plan to release an agent-managed KV-cache serving system next.

Caveats

• Pre-trained fusers are currently limited to a narrow set of Qwen3 receiver pairings.
• Multi-sharer fusion is explicitly labeled preliminary and still being worked on.
• Using custom model pairs requires manually instantiating per-layer projectors; no fully automatic cross-architecture alignment is provided.

Verdict

Worth a look if you’re building multi-agent pipelines or model ensembles and want to cut token-generation overhead. Skip it if you need broad, production-ready support for arbitrary model combinations today.