← all repositories
sybil-solutions/inference-optimisation-skill

An agent skill that hill-climbs your local LLM server's speed

Encodes a full measurement-driven optimisation workflow — ceilings, controls, gates — as a skill any coding agent can follow.

Feature · 18 Nov 2026
The Skill That Tells Your Agent When a Speed Target Defies Physics

sybil-solutions distilled its inference hill-climbs into an agent skill that computes the theoretical ceiling before anyone touches a flag — and refuses to publish numbers that can't survive a full-context soak.

Read the in-depth article →
Too new to rank. Velocity needs a few days of history.
not enough history yet
star history

What it does

It’s a SKILL.md-style agent skill (works with Claude Code, or any agent, or a human via PROMPT.md) for making a local LLM server faster: decode, prefill, speculative decoding, KV cache size, startup and time-to-first-token. The core idea is a disciplined hill-climb: compute the theoretical ceiling for your hardware first, change one lever at a time, and measure candidates against interleaved control runs so noise doesn’t fool you. Every “win” has to survive a long-context needle-retrieval gate and a concurrent full-KV soak before it counts.

The interesting bit

The ceiling math is the part most people skip. The skill forces the agent to compute bytes/token ÷ bandwidth and FLOPs/token ÷ peak up front, so it can tell you early that “100 tok/s” is physically impossible on your laptop — the README’s own B390 run reached ~55% of its decode ceiling and knew exactly where the remaining gap lived (GEMV kernel efficiency). It also ships a small engine-agnostic harness: bench.py (token counts from server usage stats, never stream chunks), ceiling.py, a serialized run queue for A/B/A interleaving, and mem.sh for real unified-memory footprints that cgroups miss.

Key highlights

  • Measured results across Intel Arc iGPU/Pro, RTX 3090 and 4× H200 — e.g. 256K prefill 297 → 628 tok/s on a laptop iGPU, and a crashed-under-load H200 setup brought to 229 tok/s decode with all six gates passing
  • Treats ±5–10% run-to-run noise as the default; single-run “wins” are assumed to be drift
  • Reference docs cover measurement traps, per-engine gotchas (vLLM, SGLang, llama.cpp, ExLlama) and hardware quirks on Intel, NVIDIA, Apple and AMD
  • Failures are logged as first-class output — PLAN.md’s list of what didn’t work is called out as most of the value
  • Standard-library Python plus bash; benchmarks any OpenAI-compatible server

Caveats

  • The headline numbers come from the authors’ own hardware and linked companion repos; you’ll need comparable measurement discipline to reproduce anything on yours
  • It’s a procedure plus a harness, not a magic flag list — expect the agent to spend real time on baselines and controls

Verdict

Worth a look if you run local models and have ever chased tok/s by randomly flipping engine flags. If you just want a one-line “make it fast” config, this will feel like homework — the good kind.

Frequently asked

What is sybil-solutions/inference-optimisation-skill?
Encodes a full measurement-driven optimisation workflow — ceilings, controls, gates — as a skill any coding agent can follow.
Is inference-optimisation-skill open source?
Yes — sybil-solutions/inference-optimisation-skill is open source, released under the MIT license.
What language is inference-optimisation-skill written in?
sybil-solutions/inference-optimisation-skill is primarily written in Python.
How popular is inference-optimisation-skill?
sybil-solutions/inference-optimisation-skill has 1.7k stars on GitHub.
Where can I find inference-optimisation-skill?
sybil-solutions/inference-optimisation-skill is on GitHub at https://github.com/sybil-solutions/inference-optimisation-skill.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.