Buoyant Unveils Open Source Control Plane for AI Workloads Running on Kubernetes

Buoyant has released an open-source control plane called Kinference, which simplifies the management of artificial intelligence (AI) workloads on top of a virtual large language model (vLLM) inference engine running on a Kubernetes cluster. The vLLM includes an inference server for managing network traffic and an inference engine that maximizes memory throughput using the PagedAttention algorithm.

This new control plane, developed based on previous work by researchers from the Sky Computing Lab at the University of California, Berkeley, enables IT teams to scale out AI workloads more cost-effectively.

The shift to self-hosted AI workloads allows organizations to eliminate dependencies on cloud services and gain better control over the quality of service (QoS) provided to each workload. Many IT teams deploy AI models in self-hosted IT environments due to security concerns, compliance requirements, and sensitivity of the data handled. However, GPU utilization rates in these environments often remain low, with a typical single-digit utilization rate, despite a global GPU shortage.

Kinference addresses this issue by providing a control plane that enables IT teams to define performance, cost, and compliance boundaries. This distribution of inference requests across the cluster, leveraging GPU cache data, queue status, and other controls, can help maximize GPU utilization. Additionally, Kinference can optionally send inference requests to cloud providers under specific conditions, such as GPU pool saturation or tenant requirements, and track GPU usage per tenant, group, or team to optimize infrastructure costs.