Run multi-node inference workloads using Ray LLM Deep Learning Containers
Learn how to deploy one large language model across two GPU nodes on Amazon EKS using the multi-node Ray LLM Deep Learning Containers, with separate prefill and decode nodes behind one OpenAI-compatible endpoint. The pre-built container ships the full inference stack, so you configure the deployment instead of assembling it.
This article explains how to run large language model (LLM) inference across multiple nodes using Amazon Elastic Kubernetes Service (Amazon EKS) and Ray LLM Deep Learning Containers. The setup consists of separate prefill and decode nodes connected by an HTTP endpoint. Separating the compute-heavy prefill phase from the memory-bandwidth-bound decode phase improves performance in workloads like Retrieval Augmented Generation, document summarization, and agentic applications.
AWS Deep Learning Containers (DLCs) simplify building the dependency chain for GPU stack compatibility, allowing customers to focus on delivering models. The Ray LLM DLC provided by AWS includes vLLM as the inference engine, Ray Serve's HTTP serving layer, NIXL for KV-cache transfer between nodes, and the Elastic Fabric Adapter (EFA) stack for high-bandwidth node-to-node communication.
The entire serving application is defined in a block of YAML, making it easy to deploy. A single helper function handles wiring the KV-cache transfer between the prefill and decode phases behind a single OpenAI-compatible endpoint.
Written by urgent.news from AWS Machine Learning's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
