The Skill That Tells Your Agent When a Speed Target Defies Physics

sybil-solutions distilled its inference hill-climbs into an agent skill that computes the theoretical ceiling before anyone touches a flag — and refuses to publish numbers that can't survive a full-context soak.
Most LLM inference optimization advice on the internet is a list of flags. Turn on FlashAttention. Try AWQ. Raise the batch size. What the flags don’t tell you is whether the number you got was good, whether it was noise, or whether the target you’re chasing was physically reachable on your hardware in the first place.
The inference-optimisation skill from sybil-solutions takes the opposite position: before you change anything, compute the ceiling. Bytes per token divided by memory bandwidth for decode. FLOPs per token divided by a realistic fraction of peak for prefill. KV cache size against the memory you actually have. Only then do you start climbing — one lever at a time, against interleaved controls, with every failure written down.
It’s an odd artifact, and the oddness is the point. It’s not a library, not a fork of vLLM, not a quantization scheme. It’s a procedure, packaged as a SKILL.md-style agent skill that Claude Code or any compatible agent can load, with a PROMPT.md fallback for models that just follow instructions. The scripts that ship with it are standard-library Python plus bash. The value isn’t in the code. It’s in the epistemics.
The boring part is where the value is
The core insight the skill encodes is unglamorous: inference benchmarking on a single machine is noisy at ±5–10%, and most single-run “wins” are drift. If you benchmark a candidate config, then a control, then another candidate, thermal state and background load will happily hand you a 10% improvement you didn’t earn. The skill’s answer is a serialized run queue that interleaves control, candidate, control — and accepts a change only if it beats both neighboring controls and regresses nothing by more than 3%.
That discipline extends to what gets measured. Token counts come from server usage statistics, never from streamed chunks, because chunk counting is a known trap. Prompt lengths come from the tokenizer, not from assumptions. Memory numbers include GPU driver residency on unified-memory devices — the footprint that cgroups can’t see, which is exactly the footprint that matters on a laptop iGPU sharing LPDDR5X with the OS. The harness flags looping output, failed requests, and — a detail that suggests hard-won experience — runs happening on battery.
None of this is clever. It’s the kind of experimental hygiene a good performance engineer applies automatically and everyone else skips. Packaging it so an AI agent applies it automatically is the actual product.
Ceilings first, because 53 tok/s means nothing without one
The skill’s signature move is making the agent compute theoretical maxima before touching a config. The README walks through why this matters using the run the skill was distilled from: an Intel Arc B390 laptop iGPU with 64 GB of unified LPDDR5X and roughly 115 GB/s of bandwidth, running a 35B-parameter MoE model at 4-bit EXL3 quantization.
The ceiling calculation said 100 tok/s single-user decode was only reachable with two useful speculative drafts. The model’s draft head accepts about 14% at depth 2, so the practical target collapsed to roughly 88–97 tok/s. The hill-climb landed at 53 tok/s median — about 55% of the practical ceiling, with the gap attributed to GEMV kernel efficiency. Prefill at 256K context reached 628 tok/s against a computed ceiling of 670–684.
This is the rare performance writeup that tells you what it didn’t achieve and why. The industry norm is to report the after number and let the reader assume it’s near-optimal. Here, the number is framed against physics: 53 tok/s is neither great nor bad in the abstract — it’s a specific fraction of a specific ceiling, and the residual gap has a named cause. The skill makes the agent do this arithmetic up front so it can tell you early, in the README’s phrasing, when a target is above the physics.
The rest of the results table follows the same pattern of measured, hardware-attributed numbers. On an Arc Pro B70, a 27B model at EXL3 4-bit hit 91.2 tok/s single-user and 365 tok/s aggregate at 16 users under vLLM with custom XPU kernels — against 25.0 tok/s for the same card running the model under llama.cpp at Q4_K_M, a gap wide enough to make engine choice the whole ballgame. An agent-session replay dropped from 193 to 147 seconds. On an RTX 3090, a 35B MoE at 3 bpw with MTP speculative decoding reached 251.7 tok/s in prose and 357.9 in code. And on 4× H200, a DeepSeek-V4.1-Flash deployment that crashed under load before the climb ended at 229.1 tok/s decode and 17,546 tok/s prefill with all six acceptance gates passing — a reminder that “before” is sometimes not slow but broken.
What the broader field says, and where this fits
The technical landscape the skill operates in is well documented elsewhere. The NVIDIA inference optimization guide and Google’s overview both describe the same two-phase structure: prefill is compute-bound and parallel, decode is memory-bound and sequential, and everything else — KV caching, PagedAttention, continuous batching, speculative decoding — is an attempt to manage that asymmetry. The KV cache history piece by Luv Bansal traces how pre-allocated contiguous caches wasted 60–80% of memory to fragmentation before vLLM’s PagedAttention applied OS-style paging and reported 2–4× throughput gains.
On speculative decoding specifically, the field has converged on a clear taxonomy: a draft model proposes, the target verifies in one pass, and the output is provably identical to standard generation — a latency optimization, not an approximation. Apple’s QuantSpec work pushes the same idea toward edge devices with hierarchical 4-bit KV caches and claims above-90% acceptance rates and up to ~2.5× speedups.
The skill’s contribution to this crowded literature is not a new technique. Its speculative-decoding guidance is a contrarian framing that cuts against how most people read the field: speculative decoding is about per-step cost, not acceptance rate. A high acceptance rate with an expensive draft is a loss; the levers that matter are pruning the draft head’s vocabulary, sweeping draft length per concurrency level and context length, and keeping sampling on the GPU. That’s the kind of guidance that only comes from reading a lot of hill-climb logs.
There’s also a structural shift this repo sits on top of: the emergence of SKILL.md files as a distribution format for expertise. A parallel example is the OpenShift AI inference-optimization skill, which packages Red Hat’s stack-specific guidance — quantization decision trees, hardware recommendations, cost comparisons — the same way. The two skills make an instructive contrast. The OpenShift one reads like vendor documentation restructured for agent consumption: authoritative, comprehensive, and anchored to a product line. The sybil-solutions one reads like a lab notebook restructured for agent consumption: opinionated, evidence-tagged, and anchored to measured runs on named hardware. Which one you’d want depends on whether you’re deploying on a managed platform or sweating a specific card on your desk.
The gates, and why they exist
The part of the procedure most likely to be skipped by anyone improvising is the end: a configuration only gets called “best” after it survives a long-context needle-retrieval gate — exact-match needles, not vibes — and a concurrent full-KV soak. Then a clean, pinned build has to be qualified from scratch and pass six acceptance gates before anything is published.
The rationale is visible in the results themselves. The B390 run quadrupled effective KV cache capacity from 264K tokens to four concurrent 256K contexts. A change like that is exactly the kind that looks great in a short benchmark and quietly degrades retrieval at the edge of the context window. Speedups that only hold for prompts you happened to test are the production gap the broader literature keeps warning about — techniques that demo well and then meet a real workload.
The honest caveats
Everything in the results table is self-reported by the organization that wrote the skill, measured on hardware they own, against ceilings they computed, with gates they defined. The raw data lives in their companion repos — local-ai-registry, trellis-serve, qwen-serve — and the contributing guidelines do ask for hardware, engine versions, before/after numbers, and links to raw data, which is more provenance than most benchmark blog posts offer. But there is no third-party replication, and the models named are niche enough that independent verification would require someone to go buy the same card.
The skill is also, in a real sense, glue. The scripts are thin; the engines — vLLM, SGLang, llama.cpp, ExLlama — do the actual work. What’s being distributed is judgment: the lever ordering, the gotchas file of known traps per engine and per vendor, the methodology notes on measurement traps, and the accumulated list of things that didn’t work. Whether an agent following the procedure reproduces the results depends on how well the prose transfers, which is an open empirical question the README doesn’t address.
And the ceiling math itself embeds assumptions — a 50%-of-peak FLOPs figure for prefill, bandwidth numbers that have to be measured rather than read off a spec sheet. The skill is honest that these are inputs, not truths.
Why it matters anyway
The economic backdrop makes the timing less mysterious. As WEKA’s analysis notes, inference is projected to be the majority of AI-optimized infrastructure spend within a few years, and Jevons paradox applies with full force: every efficiency gain increases total consumption. Meanwhile the GKE best-practices docs frame optimization as a menu of quantization methods and parallelism strategies — accurate, but silent on how to know whether you’re done.
That’s the gap this repo fills. The cloud providers will happily tell you what the levers are. Almost nobody tells you how to pull them without fooling yourself. A skill that forces an agent — or a human, since PROMPT.md works for any model — to compute the ceiling, interleave the controls, log the failures, and gate on full-context stability is doing something more durable than shipping faster kernels. It’s shipping the part of performance engineering that doesn’t go stale: the method.
The kernels will be obsoleted by the next engine release. The physics won’t.
Sources
- Mastering LLM Techniques: Inference Optimization | NVIDIA Technical Blog
- LLM Inference: Optimization, Challenges & Best Practices
- KV Caching and Speculative Decoding - The Production Gap
- What is inference optimization?
- Best practices for optimizing large language model inference with ...
- Self-Speculative Decoding with Hierarchical Quantized KV Cache
- Inference optimization - LLM Inference Handbook - Modular
- Speculative decoding: how it works & when to use it
- Inference Optimization: Faster, Cost-Effective AI
- The Complete Guide to KV Cache in LLM Inference - Medium
- openshift-ai-skills/ai-inference-optimization/SKILL.md at main
- Mastering LLM Inference Optimization From Theory to Cost ...