On-device models read the long context, cloud models get the gist

What it does

Minions is a communication protocol that pairs a small on-device language model with a frontier cloud model. The local model ingests the full long context—documents, records, or logs—and exchanges structured messages with the cloud model, which never needs to see the raw text. This design aims to reduce cloud token usage and API costs while preserving output quality. The repository implements the protocol along with demo apps, a WebGPU browser client, and an end-to-end encrypted chat variant.

The interesting bit

Instead of pre-chunking documents like a typical RAG pipeline, the local model acts as an intelligent filter that reads everything and forwards only relevant citations and summaries. The project also ships with a Secure Minions Chat mode that layers end-to-end encryption on top of this local-remote split, making the architecture useful for privacy-sensitive workloads.

Key highlights

• Local model hosts the long context; cloud model receives only distilled evidence
• Broad hardware support: ollama for CPU, lemonade for AMD, tokasaurus for NVIDIA, plus Cartesia-MLX on Apple Silicon
• WebGPU demo runs local inference entirely in the browser with no server setup
• Secure Minions Chat adds end-to-end encryption to the local-cloud split
• Backed by a Stanford Hazy Research paper and blog posts

Caveats

• The repository is explicitly framed as a research demonstration, not a production framework
• Backend coverage is uneven; lemonade currently does not support the Minion-CUA protocol

Verdict

Experiment with it if you are prototyping cost-sensitive or privacy-conscious LLM applications that need to process long contexts. Look elsewhere if you need a mature, batteries-included orchestration framework.