← all repositories
ashhart/TensorFold

Speculative decoding that can't change your answer

TensorFold serves a short list of open model families on Apple Silicon and NVIDIA GPUs behind an OpenAI-compatible API, and accepts a drafted token only when it equals what the same engine would emit serially.

TensorFold
Velocity · 7d
+93
★ / day
Trend
↗accelerating
star history

What it does

TensorFold is a self-hosted, OpenAI-compatible server for a fixed set of model families — Nemotron 3.5 Lightning, Qwen3.8-27B, Qwen3.8 Flash Next, GLM-5.3-Flash, and Gemma 4 — running on MLX for Apple Silicon or CUDA for NVIDIA GPUs. Each family supplies its own kernels and draft-verification code, so this is less a general inference engine and more a set of hand-fitted, per-family implementations behind one /v1 endpoint. Drafting comes from included MTP heads, from copies of the model’s own context, or from optional separate draft models.

The interesting bit

The exactness contract. A drafted token is accepted only if it equals the token the same engine would have produced serially, with sampling keyed to the prompt or seed, absolute position, and token ID — and you can audit any request against the slow path by resending it with "draft": false. The MLX engine even shares compute rounds across concurrent requests while keeping per-stream state, and load-time checks disable sharing wherever a family can’t reproduce its serial arithmetic. Exactness here is checkable, not taken on faith.

Key highlights

  • OpenAI-compatible endpoint with model IDs from /v1/models; --backend picks MLX or CUDA, and auto defaults to MLX on macOS.
  • Per-family drafting: built-in MTP heads, self-drafting from context copies, or optional DFlash2 draft models; --no-drafts forces the serial reference path.
  • The unglamorous core is memory governance: admission accounts for weights, cache growth, reply tokens, and prefill workspace; evicted conversation prefixes can spill to disk and be read back instead of re-prefilling; the default process budget is 70% of RAM (85% for GLM on smaller Macs).
  • Quantization breadth: Qwen3.8-27B reads MLX affine 2- through 8-bit checkpoints, including mixed layer formats, plus experimental EXL3 support on CUDA.
  • Two-rank CUDA execution for GLM, and Flash Next’s --kv-dtype int8/int4 shrinks the KV cache so the same memory admits a longer window.

Caveats

  • The memory-class table is entirely TBD — the README is explicit these are qualification slots, not measurements, pending a public prompt fixture; no throughput or latency numbers are published either.
  • Exactness is scoped to the same engine, weights, runtime, and settings — it does not mean MLX and CUDA, or different quantizations, agree with each other.
  • Coverage is a closed list of families and specific checkpoints; many options are MLX-only, CUDA serves GLM and Nemotron one request at a time, the new packed row readers are pending hardware qualification, and GLM’s optional DFlash2 draft checkpoint carries non-commercial license terms.

Verdict

Worth a look if you run one of the supported families on a Mac or CUDA box and want an OpenAI endpoint where acceleration provably doesn’t alter tokens. Skip it if you need broad model coverage, cross-backend reproducibility, or published benchmarks — the README hasn’t filled those in yet, and says so.

Frequently asked

What is ashhart/TensorFold?
TensorFold serves a short list of open model families on Apple Silicon and NVIDIA GPUs behind an OpenAI-compatible API, and accepts a drafted token only when it equals what the same engine would emit serially.
Is TensorFold open source?
Yes — ashhart/TensorFold is open source, released under the Apache-2.0 license.
What language is TensorFold written in?
ashhart/TensorFold is primarily written in Zig.
How popular is TensorFold?
ashhart/TensorFold has 2.4k stars on GitHub and is currently accelerating.
Where can I find TensorFold?
ashhart/TensorFold is on GitHub at https://github.com/ashhart/TensorFold.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.