A 321B model on your gaming rig, and it actually moves

What it does

Maya is a single-model inference engine, written in C++, built specifically for GLM-5.3-Flash (321B parameters, ~18B active per token, up to 1M context). It serves the model through a browser chat with a live hardware monitor, plus OpenAI- and Anthropic-compatible APIs with tool calls — so coding agents like Claude Code can point straight at it. Nothing leaves your machine.

The interesting bit

The whole trick is expert-tiering: the MoE experts your conversation uses most live on the GPU, the next tier in pinned RAM, the rest on NVMe, shuffled as you chat. The engine also profiles your CPU, RAM, and SSD and tunes itself to the machine — a benchmark table full of real user numbers (118 tokens/s on four RTX 4090s, 30+ on a single RTX 5090) backs the claim.

Key highlights

• Custom quants from 96.5 GB (Maya-S) to 156.3 GB (Maya-L), retaining 97.7–99.2% of the FP8 model’s zero-shot accuracy
• Up to 8 concurrent requests, with measured throughput gains (up to ~2x on hardware that holds all experts in VRAM)
• KV-cache streaming keeps attention in RAM so VRAM fits more experts — 1M context on a single RTX 3090
• Optional DFlash2 drafter for speculative decoding (+44% on a nine-GPU Maya-L setup), kept only where calibration shows it helps
• Runs on NVIDIA (V100 onward, up to 16 cards) and, experimentally, AMD ROCm and Windows

Caveats

• AMD and Windows support are explicitly experimental; AMD is text-only
• The engine pins nearly all free RAM while running, so other programs starve
• The DFlash2 drafter carries a non-commercial license (CC BY-NC-ND 4.0)

Verdict

If you have a workstation with a fast NVMe drive and want a frontier-sized model fully under your own roof, this is one of the more credible attempts. If you’re hoping to swap models weekly, look elsewhere — Maya is married to exactly one.