sybil-solutions · Inference Optimisation Skill

inference-optimisation

An agent skill for hill-climbing the speed of a local LLM server across decode, speculative decoding, prefill, KV cache, startup and time to first token, under a memory cap. It works with Claude Code and any agent that loads SKILL.md-style skills, and includes a benchmark harness (bench.py, ceiling.py, queue.sh). The README reports measured gains on Intel Arc B390, Arc Pro B70, RTX 3090 and 4× H200 NVL, such as Arc B390 decode for one user rising from 43.6 to 53 tok/s. Licence is not stated on the page.

Open the source ↗

Similar Developer Tools releases

Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.