Multi-Region training with Amazon SageMaker HyperPod and Qumulo

Amazon SageMaker HyperPod and Qumulo enable you to place training compute in one AWS Region while keeping your dataset in another. Training large AI models demands significant GPU capacity, yet compute resources and training data may not always be located in the same AWS Region. Accessing data across regions introduces network latency and transfer costs, forcing teams to choose between replicating petabytes of data or enduring cross-region latency.

This pairing addresses this trade-off, allowing teams to maintain up-to-date frontier models without data movement or throughput sacrifices.

In this post, a solution to this challenge is presented, including its architecture and results from a cross-region training run. By pairing Amazon SageMaker HyperPod with Cloud Native Qumulo (CNQ) and its Cloud Data Fabric (CDF), training jobs can access datasets stored in another AWS Region or on-premises without copying data, altering code, or sacrificing throughput.

Validation tests were conducted by running the same training job independently on two clusters. The hub cluster resided in the US East (Ohio) Region (us-east-2), where the data was also stored. The spoke cluster, located in the US West (Oregon) Region (us-west-2), accessed the data remotely via CDF with 60 ms of network latency.

After a brief warmup, the spoke cluster matched the hub's performance, demonstrating that pairing Amazon SageMaker HyperPod with CNQ provides two key benefits: performance equal to single-region training and optimal utilization in remote clusters.