sybil-solutions · Inference Optimisation Skill
inference-optimisation
An agent skill for hill-climbing the speed of a local LLM server across decode, speculative decoding, prefill, KV cache, startup and time to first token, under a memory cap. It works with Claude Code and any agent that loads SKILL.md-style skills, and includes a benchmark harness (bench.py, ceiling.py, queue.sh). The README reports measured gains on Intel Arc B390, Arc Pro B70, RTX 3090 and 4× H200 NVL, such as Arc B390 decode for one user rising from 43.6 to 53 tok/s. Licence is not stated on the page.
Open the source ↗Similar Developer Tools releases
- Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
AWS Machine Learning Blog (Amazon Web Services) · Managed Dynamo inference · Week 48, 2026
- SageMaker AI KV cache aware routing
Amazon Web Services (AWS) · KV cache routing · Week 48, 2026
- Together Compiler
Together AI · Agent-driven GPU compiler · Week 48, 2026
- cuBLAS Ozaki-II and 2M FP64 Algorithms
NVIDIA · FP64 GEMM Emulation · Week 47, 2026
- TPU Sync
Google · KV Cache Transfer · Week 47, 2026
- SGLang-Neuron
RadixArk with Annapurna Labs (AWS) · Trainium inference plugin · Week 47, 2026
Published by Quanta AI Labs, an AI agency in India. Browse every week in AI News Weekly.