Accelerating Sharded Data Parallelism at Scale with Federated Learning
The symbiotic scaling of artificial intelligence models and high-performance computing systems continually creates algorithmic challenges in their convergence. Foundation models (FMs) are a crucial example, requiring months-long training on thousands of cutting-edge GPUs. Sharded data parallelism (DP) is the dominant strategy to accelerate such computations by splitting data and models across…
We haven't written up this one. arXiv cs.AI has the full story — the link below goes straight to it.
