IBM-backed llm-d framework adds controls for running AI models in-house
IBM Corp. and several other contributors to the open-source llm-d project are releasing version 1.0 of their distributed artificial intelligence inference framework, adding operational controls intended to help enterprises run large models reliably on their own infrastructure. The release, announced ahead of the PyTorch Conference North America in San Jose today and tomorrow, focuses on […] The…
IBM has released version 1.0 of the llm-d framework, an open-source distributed artificial intelligence inference framework that adds operational controls for running AI models in-house. The release, announced ahead of the PyTorch Conference North America, focuses on managing competing workloads, maintaining service during upgrades, and improving hardware efficiency.
The framework, built for Kubernetes, coordinates model serving across computing infrastructure and aims to help enterprises reduce dependence on a single model provider, cloud, or hardware platform. IBM, Red Hat, and Google are the lead maintainers of the project, which was co-founded with CoreWeave Inc. and Nvidia Corp. Other contributors include AMD, Cisco, Hugging Face, Intel, and Mistral AI SAS.
The release adds support for quality of service across multiple tenants, zero-downtime operations, and serving patterns tailored to AI agents, with flow controls and prioritization designed to prevent bursts of agent activity from overwhelming services that require fast responses.
Brief written by urgent.news from SiliconANGLE's own syndicated text. Machine-written — may contain errors; check the original before relying on it.
