Speculative decoding that can't change your answer
TensorFold serves a short list of open model families on Apple Silicon and NVIDIA GPUs behind an OpenAI-compatible API, and accepts a drafted token only when it equals what the same engine would emit serially.

What it does
TensorFold is a self-hosted, OpenAI-compatible server for a fixed set of model families — Nemotron 3.5 Lightning, Qwen3.8-27B, Qwen3.8 Flash Next, GLM-5.3-Flash, and Gemma 4 — running on MLX for Apple Silicon or CUDA for NVIDIA GPUs. Each family supplies its own kernels and draft-verification code, so this is less a general inference engine and more a set of hand-fitted, per-family implementations behind one /v1 endpoint. Drafting comes from included MTP heads, from copies of the model’s own context, or from optional separate draft models.
The interesting bit
The exactness contract. A drafted token is accepted only if it equals the token the same engine would have produced serially, with sampling keyed to the prompt or seed, absolute position, and token ID — and you can audit any request against the slow path by resending it with "draft": false. The MLX engine even shares compute rounds across concurrent requests while keeping per-stream state, and load-time checks disable sharing wherever a family can’t reproduce its serial arithmetic. Exactness here is checkable, not taken on faith.
Key highlights
- OpenAI-compatible endpoint with model IDs from
/v1/models;--backendpicks MLX or CUDA, andautodefaults to MLX on macOS. - Per-family drafting: built-in MTP heads, self-drafting from context copies, or optional DFlash2 draft models;
--no-draftsforces the serial reference path. - The unglamorous core is memory governance: admission accounts for weights, cache growth, reply tokens, and prefill workspace; evicted conversation prefixes can spill to disk and be read back instead of re-prefilling; the default process budget is 70% of RAM (85% for GLM on smaller Macs).
- Quantization breadth: Qwen3.8-27B reads MLX affine 2- through 8-bit checkpoints, including mixed layer formats, plus experimental EXL3 support on CUDA.
- Two-rank CUDA execution for GLM, and Flash Next’s
--kv-dtype int8/int4shrinks the KV cache so the same memory admits a longer window.
Caveats
- The memory-class table is entirely TBD — the README is explicit these are qualification slots, not measurements, pending a public prompt fixture; no throughput or latency numbers are published either.
- Exactness is scoped to the same engine, weights, runtime, and settings — it does not mean MLX and CUDA, or different quantizations, agree with each other.
- Coverage is a closed list of families and specific checkpoints; many options are MLX-only, CUDA serves GLM and Nemotron one request at a time, the new packed row readers are pending hardware qualification, and GLM’s optional DFlash2 draft checkpoint carries non-commercial license terms.
Verdict
Worth a look if you run one of the supported families on a Mac or CUDA box and want an OpenAI endpoint where acceleration provably doesn’t alter tokens. Skip it if you need broad model coverage, cross-backend reproducibility, or published benchmarks — the README hasn’t filled those in yet, and says so.
Frequently asked
- What is ashhart/TensorFold?
- TensorFold serves a short list of open model families on Apple Silicon and NVIDIA GPUs behind an OpenAI-compatible API, and accepts a drafted token only when it equals what the same engine would emit serially.
- Is TensorFold open source?
- Yes — ashhart/TensorFold is open source, released under the Apache-2.0 license.
- What language is TensorFold written in?
- ashhart/TensorFold is primarily written in Zig.
- How popular is TensorFold?
- ashhart/TensorFold has 2.4k stars on GitHub and is currently accelerating.
- Where can I find TensorFold?
- ashhart/TensorFold is on GitHub at https://github.com/ashhart/TensorFold.