The 982-Gigabyte Model on a 64-Gigabyte Machine

The Audacity of Local Inference

There is a certain kind of engineering project that announces itself with numbers so large they sound like a typo. WASTE—the Weight-Aware Streaming Tensor Engine—currently runs Kimi K3, a 2.78-trillion-parameter mixture-of-experts model, on a 64 GB MacBook Pro. The converted model container occupies 982 GB of NVMe storage. Throughput is roughly 0.6 tokens per second. This is the full, undistilled, unpruned model, not a quantized shadow of itself. The developers at SQLite Cloud have essentially asked whether the boundary between “cloud-only” and “local-capable” is a hardware limit or a software assumption.

The answer, so far, is that it is mostly an assumption. WASTE is written in C, carries no third-party runtime dependencies, and requires neither CUDA nor BLAS for its CPU inference path. It is intentionally narrow: a probe designed to discover how far local inference can stretch when model weights live on fast storage rather than in RAM. The name itself is a mission statement. The team argues that every token answered by a cloud service is paid for twice—once on the invoice and again in datacenter electricity—when the model could, “barely, awkwardly, but genuinely,” run on hardware already sitting on a desk. The acronym came second.

What makes the project particularly resonant right now is that it arrives at a moment when the AI industry is obsessed with centralization. Frontier models are trained on massive clusters and served from massive clusters. WASTE offers a contrarian data point: a trillion-parameter model executing on a laptop, albeit slowly. It also arrives alongside a growing body of research into memory-hierarchy offloading, from academic I/O characterizations of SSD-based model serving to industry experiments in CPU-GPU memory sharing. The difference is that WASTE is not a simulation; it is a shipping C library that already exposes an OpenAI-compatible chat API and processes multimodal prompts.

Storage as Memory, Memory as Cache

The technical insight behind WASTE is architectural, not magical. Kimi K3 is a mixture-of-experts model, meaning that while its parameter count is measured in trillions, only about four percent of them are active for any given token. The engine exploits this sparsity by keeping the shared “trunk” of the model resident in memory—about 29 GB at 4K context—and treating the rest as a streaming library. Selected experts are read directly from disk into a bounded RAM cache, one aligned read per expert, with reads overlapped against computation. A lookahead router predicts which experts the next layer will need and begins fetching them early, though the real router retains final authority so that timing changes without altering results.

This inverts the usual memory hierarchy. In conventional inference, RAM is the primary residence and disk is the last resort. WASTE makes disk the primary store and RAM a cache. The on-disk container is arranged precisely for this: experts are laid out to minimize seek latency, and the engine relies on the sustained sequential throughput of internal NVMe storage—measured at 12.78 GB/s on the test machine—rather than random access. A cold token pulls roughly 17 GB of expert data off the SSD. The project’s documentation notes that a USB enclosure managing 0.94 GB/s is insufficient; the interface is the bottleneck.

The compression strategy is similarly pragmatic. Experts are stored with 3-bit residual vector quantization, while the more sensitive shared weights remain at 4 or 8 bits. The KV cache, often the hidden memory killer in long-context inference, is compressed through Kimi Delta Attention to roughly 0.21 GB at 4K context instead of the 11.25 GB it would otherwise consume. These are not independent tricks; they are cumulative concessions to the reality that a 64 GB machine cannot hold a 982 GB model, but it might be able to stream it.

The Cache Cliff and Other Counterintuitions

WASTE’s performance measurements contain a warning that every systems engineer should study. On the test machine, the default 17.56 GB expert cache yields a 36.2 percent hit rate and the peak throughput of 0.63 tok/s. Shrinking the cache to 3.32 GB drops the hit rate to 29.1 percent and costs only about ten percent of throughput. But expanding the cache to 23.32 GB or 29.32 GB—still inside the process memory budget—causes throughput to collapse to 0.07–0.09 tok/s, an eightfold regression. The hit rate climbs, the bytes read fall, and the machine still slows to a crawl.

The reason is subtle and brutal: once the expert cache grows too large, the operating system begins paging resident memory. A cache hit becomes a page fault. The engine is technically within its budget, but the machine is not. This is the kind of result that survives only in a project willing to publish negative findings alongside positive ones. The team documents failed ideas and discarded hypotheses deliberately, treating measurements as experimental results rather than marketing figures. It is a refreshing stance in a field that often rounds up.

Not Edge AI, But Personal Frontier

WASTE sits awkwardly in the current taxonomy of efficient inference. It is not TinyML. The edge-AI literature—focused on microcontrollers, kilobyte-scale memory, and sub-watt power budgets—represents the opposite end of the spectrum. Projects in that space prune and quantize models down to insect size so they fit on sensors. WASTE does the opposite: it preserves a frontier-scale model intact and asks the desktop to grow around it.

In that sense, it shares more DNA with the emerging industry interest in memory-hierarchy offloading. NVIDIA has explored CPU-GPU memory sharing for large-scale inference, and local-LLM communities have sustained active curiosity about NVMe offloading as a practical path for consumer hardware. WASTE is a concrete existence proof in that conversation, demonstrating that a trillion-parameter model can be served from a single workstation without a datacenter footprint. It also supports multimodal inference—images are expanded into prompt positions and fed through both a vision tower and the language model—though this is currently expensive, with a single 896×896 image taking about 15.7 seconds in the vision tower alone.

The project also carries a meta-narrative about how it was built. The ideas, hypotheses, tests, and decisions are human; the code is written by large language models. At this scale, the team argues, that is the only way to iterate on new algorithms fast enough. Whether this is a scalable methodology or a temporary artifact of the current moment, it is a notable admission. WASTE is not just running a big model locally; it is running a big model locally so that it can eventually improve itself, with the team currently using Opus 5 for extra thinking capacity.

Outlook: The Storage Wall

For all its ambition, WASTE is still a research vehicle. Zero-point-six tokens per second is usable for batch processing or for proving a point, but it is not a conversational interface. A 32 GB machine can open the model but will page heavily. The format and API are explicitly not frozen, and the software is described as changing very quickly.

The open questions are architectural. CUDA, Metal, and other hardware-specific backends remain unexplored and may provide significant gains. The team tracks these as future directions rather than promises. The deeper tension is whether storage bandwidth can continue to substitute for RAM capacity as models grow. Today, WASTE proves that a 2.78-trillion-parameter model can execute on a single consumer workstation. Tomorrow, the question is whether that execution can become fast enough to be practical, or whether it will remain a boundary-pushing experiment—an existence proof that the cloud is not the only place for a frontier mind.