{
  "id": 20961928,
  "title": "Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters",
  "url": "https://urgent.news/2026/11/20/introducing-amazon-sagemaker-hyperpod-global-inference-gateway-one",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-20T22:35:24.000Z",
  "source": {
    "name": "AWS Machine Learning",
    "slug": "aws-machine-learning",
    "url": "https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemaker-hyperpod-global-inference-gateway-one-endpoint-across-regions-and-clusters/"
  },
  "original_language": "en",
  "account": "During periods of high demand, a request may become stuck in a queue on an overburdened HyperPod cluster, while GPU resources on another cluster, potentially in a different AWS Region, remain idle and underutilized. The Inference Gateway rectifies this issue by assigning each HyperPod cluster a single, GPU-aware endpoint that reads real-time signals like key-value cache utilization and queue depth to direct every request to the most suitable pod. This solution addresses routing within a single cluster; however, it does not cover routing across clusters. To tackle this challenge, Amazon SageMaker HyperPod now introduces the Global Inference Gateway, a single SageMaker-managed gateway that routes requests across all registered clusters, whether they are in the same Region or across multiple Regions. The gateway selects the appropriate cluster and transports the request via the AWS backbone, complete with automatic failover and authentication using AWS Signature Version 4 (SigV4) and AWS Identity and Access Management (IAM). Within each cluster, the Inference Gateway continues to identify the optimal pod, as it has always done. The Global Inference Gateway constitutes the second layer of the architecture previously described. The task at hand is to manage capacity spread across multiple clusters and Regions. Models have been growing in size, making it increasingly difficult to locate sufficient GPU capacity. To ensure adequate resources, developers are spreading these clusters and Regions. For instance, imagine a team serving four models across five HyperPod clusters in two Regions. Without the Inference Gateway, clients would need to manage 20 endpoints: one per model per cluster. The Inference Gateway simplifies this to five endpoints, one per cluster. Nevertheless, each client must still select a cluster, which has its drawbacks: uneven load, manual failover, and the need for custom networking infrastructure. The Global Inference Gateway provides one endpoint that directs each request to the most suitable cluster and autonomously diverts traffic from a failed cluster. The key benefits of this solution are its ability to extract more value from existing GPU capacity with reduced operational overhead. It enables the utilization of idle GPUs during peak times, automatically redirects traffic to the next-best cluster in case of a cluster failure, and allows clients to utilize a single endpoint for all requests, eliminating the need to track multiple endpoints. Moreover, it eliminates the necessity for custom networking, such as load balancers, DNS, and cross-Region networking. Security is maintained by default, with every request authenticated using SigV4 and IAM, and all traffic routed over AWS PrivateLink. A customer, Bagelcode, shared their experience with this new feature during beta testing, highlighting how the Global Inference Gateway in Amazon SageMaker HyperPod routed requests across two AWS Regions through a single endpoint, ensuring continuous service despite one cluster's inability to serve the model. This capability provided cross-Region failover without requiring the platform team to build or maintain a separate routing layer. The setup of the gateway involves two decisions: selecting the clusters to open for routing and determining which clusters to group behind the gateway. The publication of a cluster is a one-time process that involves turning on the spec.globalRouting block within the cluster's existing InferenceGatewayConfig. The HyperPod Inference controller then privately publishes the cluster from your account. Creating the gateway is also a one-time operation that involves calling CreateGlobalInferenceGateway with the clusters to route across, resulting in a single SageMaker-managed endpoint. Each cluster must be published before it can be included in a gateway; otherwise, the operation will fail and provide instructions on how to resolve the issue. The flow of a request through the Global Inference Gateway is illustrated in Figure 1: (1) the client signs the request with SigV4 and sends it to the gateway's regional endpoint, (2) the gateway authenticates the caller and selects the optimal cluster, (3) traffic is carried across the AWS backbone using PrivateLink, either within the same Region or to a different Region, and (4) the cluster's Inference Gateway routes the request to the appropriate model pod. AWS manages the rest of the infrastructure, including the routing fleet, private connectivity, certificate and validation handling, health checks, failover mechanisms, authentication protocols, and rate limits.",
  "summary": "Amazon SageMaker HyperPod introduces the Global Inference Gateway, one managed endpoint that routes inference requests across your HyperPod clusters in one Region or across Regions. It picks the best cluster, fails over automatically, and authenticates every request with SigV4 and IAM over AWS PrivateLink.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}