← all repositories
incoai/splash

4.5× llama.cpp on a Mac, by refusing to be a general engine

A Mac-native inference engine for coding agents that buys 2.5–5.3× decode speed over llama.cpp by being deliberately per-model.

splash
Too new to rank. Velocity needs a few days of history.
not enough history yet
star history

What it does Splash is a local LLM server for Apple-silicon Macs that speaks the APIs tools already use — OpenAI Chat Completions, Responses, and Completions, plus Anthropic Messages — with streaming, tool calls, JSON Schema output, images, and inline PDFs. Coding agents like Claude Code, Codex, OpenCode, Hermes, and Pi can run against it, and there’s a built-in chat page with vision support. Memory and context length are planned automatically up to the model’s native window, cached prefixes are reused across requests, and concurrent requests are batched.

The interesting bit llama.cpp is a generalist: it will run nearly anything you hand it in GGUF. Splash bets the other way — each supported model ships with a trained DFlash2 speculative-decoding draft and Metal kernels matched to its exact shapes, with only the runtime, scheduler, and cache shared between them. Speculative decoding is the old trick (a small draft proposes tokens, the big model verifies them in one pass), but per-model training is what turns it into 74 tok/s on a 27B model where llama.cpp does 16 — same weights, same machine.

Key highlights

  • On identical Unsloth Q4_K_M weights on Metal: 74 vs 16 tok/s for the 27B and 175 vs 69 for the 35B-A3B on an M5 Pro; 92 vs 17 and 209 vs 66 on an M3 Max — 2.5–5.3× overall, 2.7–4.6× even against llama.cpp’s MTP mode
  • Prefix caching brings a 32K-prompt replay to first token in 123–282 ms, 6.6–7.3× the next-fastest engine in their benchmark
  • Concurrent requests batch automatically: 170 tok/s aggregate across four simultaneous short prompts on the 27B (3.9×)
  • Memory and context are sized automatically; KV cache defaults to 8-bit INT8, with BF16 and optional SSD offload (--max-cache-disk)
  • Kernels ship precompiled — no Xcode, no per-machine tuning; weights are prepared once and mapped from disk

Caveats

  • Model support is deliberately narrow: two Qwen families (Qwen3.8-27B, Qwen3.6-35B-A3B) plus Prism’s Ternary Bonsai 2; UD-Q8_K_XL and BF16 GGUF targets are excluded
  • The hardware floor is high — M3 or newer, macOS 26.4+, and 36 GB of unified memory for the 4-bit models (48 GB recommended); 24 GB Macs must drop to smaller variants
  • Output isn’t bit-identical to llama.cpp: next-token agreement measured at 99.30–99.45% on the 27B and 97.83–98.14% on the 35B, so long generations can drift — and all these benchmarks are the project’s own

Verdict If you run coding agents on a recent, memory-rich Mac and one of the two supported Qwens is your model, this is a serious upgrade over llama.cpp. Older hardware or a different model family, and the generalists still have you covered.

Frequently asked

What is incoai/splash?
A Mac-native inference engine for coding agents that buys 2.5–5.3× decode speed over llama.cpp by being deliberately per-model.
Is splash open source?
Yes — incoai/splash is open source, released under the Apache-2.0 license.
What language is splash written in?
incoai/splash is primarily written in C++.
How popular is splash?
incoai/splash has 2.4k stars on GitHub.
Where can I find splash?
incoai/splash is on GitHub at https://github.com/incoai/splash.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.