An agent skill that hill-climbs your local LLM server's speed
Encodes a full measurement-driven optimisation workflow — ceilings, controls, gates — as a skill any coding agent can follow.
sybil-solutions distilled its inference hill-climbs into an agent skill that computes the theoretical ceiling before anyone touches a flag — and refuses to publish numbers that can't survive a full-context soak.
What it does
It’s a SKILL.md-style agent skill (works with Claude Code, or any agent, or a human via PROMPT.md) for making a local LLM server faster: decode, prefill, speculative decoding, KV cache size, startup and time-to-first-token. The core idea is a disciplined hill-climb: compute the theoretical ceiling for your hardware first, change one lever at a time, and measure candidates against interleaved control runs so noise doesn’t fool you. Every “win” has to survive a long-context needle-retrieval gate and a concurrent full-KV soak before it counts.
The interesting bit
The ceiling math is the part most people skip. The skill forces the agent to compute bytes/token ÷ bandwidth and FLOPs/token ÷ peak up front, so it can tell you early that “100 tok/s” is physically impossible on your laptop — the README’s own B390 run reached ~55% of its decode ceiling and knew exactly where the remaining gap lived (GEMV kernel efficiency). It also ships a small engine-agnostic harness: bench.py (token counts from server usage stats, never stream chunks), ceiling.py, a serialized run queue for A/B/A interleaving, and mem.sh for real unified-memory footprints that cgroups miss.
Key highlights
- Measured results across Intel Arc iGPU/Pro, RTX 3090 and 4× H200 — e.g. 256K prefill 297 → 628 tok/s on a laptop iGPU, and a crashed-under-load H200 setup brought to 229 tok/s decode with all six gates passing
- Treats ±5–10% run-to-run noise as the default; single-run “wins” are assumed to be drift
- Reference docs cover measurement traps, per-engine gotchas (vLLM, SGLang, llama.cpp, ExLlama) and hardware quirks on Intel, NVIDIA, Apple and AMD
- Failures are logged as first-class output —
PLAN.md’s list of what didn’t work is called out as most of the value - Standard-library Python plus bash; benchmarks any OpenAI-compatible server
Caveats
- The headline numbers come from the authors’ own hardware and linked companion repos; you’ll need comparable measurement discipline to reproduce anything on yours
- It’s a procedure plus a harness, not a magic flag list — expect the agent to spend real time on baselines and controls
Verdict
Worth a look if you run local models and have ever chased tok/s by randomly flipping engine flags. If you just want a one-line “make it fast” config, this will feel like homework — the good kind.
Frequently asked
- What is sybil-solutions/inference-optimisation-skill?
- Encodes a full measurement-driven optimisation workflow — ceilings, controls, gates — as a skill any coding agent can follow.
- Is inference-optimisation-skill open source?
- Yes — sybil-solutions/inference-optimisation-skill is open source, released under the MIT license.
- What language is inference-optimisation-skill written in?
- sybil-solutions/inference-optimisation-skill is primarily written in Python.
- How popular is inference-optimisation-skill?
- sybil-solutions/inference-optimisation-skill has 1.7k stars on GitHub.
- Where can I find inference-optimisation-skill?
- sybil-solutions/inference-optimisation-skill is on GitHub at https://github.com/sybil-solutions/inference-optimisation-skill.