Sub-15 ms structured decisions without generating a single token

What it does laya-mlx is a native MLX runtime for Laya typed-decision models on Apple Silicon. You give it a state and structured questions—multiple choice, rubric scores, or true/false propositions—and it returns probabilities from a single bidirectional forward pass. After the initial weight download, everything runs offline with no PyTorch, Transformers, or cloud API.

The interesting bit Instead of treating an LLM as a text generator and parsing its output, the project uses a bidirectional encoder—ModernBERT-large or mmBERT-base—with task-specific decision heads to emit structured answers directly. The result is median end-to-end latency of 7.4 ms for short multilingual questions and 13.4 ms for English, demonstrated by a real-time Snake terminal game that runs at 75 moves per second with a visible safety-correction layer.

Key highlights

• Zero output tokens: decisions come from encoder hidden states, not autoregressive decoding.
• Validated parity: all 63 validation fixtures matched upstream across both FP32 and FP16, totaling 378/378 passing comparisons.
• Speed: 7.39 ms median (multilingual) and 13.42 ms median (English) on M3 Max; batched throughput reaches 395 questions per second.
• Fully local: pure MLX inference with pre-converted Hugging Face checkpoints; no generative runtime required.
• Router included: built-in language routing and checkpoint selection, with optional resident-model preloading for mixed workloads.

Caveats

• Apple Silicon and macOS only; requires macOS 14+ and Python 3.11+.
• Inference and conversion only—training and fine-tuning remain in the upstream project.
• BF16 is available but sits outside the published validation matrix, and probabilities can drift slightly across precisions even when the selected label agrees.

Verdict Reach for this if you need low-latency, structured-decision scoring on a Mac and would rather have a fast encoder than a chatty LLM. Look elsewhere if you need generative text, cross-platform deployment, or model training.