Core ML ports that missed their 10× goal—and published it

What it does Laya-CoreML is an independent port of the Laya decision model to Apple Core ML. It runs open-weight models on Apple Silicon via CPU, GPU, or the Neural Engine to return structured probabilities for choice, ordinal score, and boolean noul questions—no autoregressive decoding, no generated JSON to parse. The package includes a terminal Snake demo that sustains roughly 50 decisions per second while rendering live model probabilities and safety interventions.

The interesting bit Most porting projects trumpet their wins and bury the losses; this one publishes detailed benchmark notebooks showing it fell short of its own 10× speedup target, settling for roughly 1.4× faster inference and 2.8× better energy efficiency than compiled MLX on an M3 Max. The authors also document which quantization experiments failed their fidelity gates and ship a W8 palette-compressed variant that barely edges out FP16, which is the kind of thoroughness you want in infrastructure you might actually deploy.

Key highlights

• ~4.98 ms P50 latency for short decisions on M3 Max ANE FP16; an approximate W8 variant reaches 4.88 ms
• 2.78× better system energy per decision than compiled MLX FP16, measured via direct SMC sensor readings
• Validated port fidelity: 189/189 matching answers on general FP16 checkpoints, and 59/59 on ANE FP16 with maximum calibrated-probability drift under 0.003
• Inference requires no PyTorch, Transformers, or MLX; runs fully offline after initial model download
• ANE bundles are strictly limited to 96 tokens (B1/L96); longer contexts require falling back to CPU+GPU checkpoints with up to 1024 tokens

Caveats

• The 96-token ANE capacity limit is hard: exceed it and you get a capacity error, not a graceful fallback
• The promised 10× speed-and-energy improvement over MLX was not achieved; real gains are modest and honestly reported
• Six- and four-bit quantization experiments failed the project’s fidelity gates and remain unpublished

Verdict Worth exploring if you need deterministic, low-latency structured decisions on Apple Silicon without pulling in PyTorch or MLX. Not the right tool if you need long-context reasoning or generative output.