Buoyant · Inference Control Plane

Kinference

Kinference is an open-source inference control plane for Kubernetes and vLLM, released by Buoyant, the company behind Linkerd. It sits in front of vLLM workers and routes requests using declarative policies, KV-cache-aware load balancing and per-tenant metrics. In self-reported benchmarks on 3 L40S GPUs, median time-to-first-token on shared-prefix traffic fell from 6.88 s to 0.31 s versus plain Kubernetes Service routing, with no gain on prompts without shared prefixes. It is Apache 2.0 licensed and still alpha (v0.1.0-alpha.7).