Beyond Kubernetes at Modal: How to Scale 1 Million Concurrent Sandboxes in Seconds
In a recent article, Colin Weld and Connor Adams, staff engineers at Modal, describe how they rebuilt their sandbox infrastructure from the ground up to support millions of concurrent sandboxes and tens of thousands of sandbox creations per second. By Sergio De Simone
Modal, a serverless compute platform designed for AI workloads, has developed a novel approach to scaling sandbox infrastructure to support 1 million concurrent sandboxes in mere seconds. Traditional container orchestration systems like Kubernetes face significant challenges at this scale due to their reliance on centralized coordination and strongly consistent state.
The sheer number of containers and the need for numerous compute nodes cause the load on scheduling algorithms and the central durable store (etcd) to grow with the number of nodes and pods, leading to scaling limits.
Weld and Adams, staff engineers at Modal, opted to eliminate global coordination and enable each worker to act as its own source of truth. They replaced the single, serialized scheduler with a fleet of scheduling servers operating in parallel, allowing the scheduling layer to scale horizontally. Once a scheduling server assigns a sandbox to a worker, it directly requests the sandbox creation via RPC.
Workers accept or reject the request based on available resources. This architecture has only one bottleneck: all workers publish their state as a single Redis stream.
In a benchmark, Modal created 1 million sandboxes in under a minute, with a median startup-to-code time of less than 0.5 seconds. AWS principal AI engineer Alex Jones highlighted that Modal's success lies in its decision to bypass Kubernetes and address its inherent limitations. Jones believes this represents a significant shift away from extending Kubernetes and towards a new approach to coordination and execution.
The execution plane in Modal requires isolated boundaries that appear in milliseconds, while the coordination plane still benefits from Kubernetes' strengths. This innovative approach demonstrates that Kubernetes may not be adapting swiftly enough to meet the demands of GenAI infrastructure, paving the way for a decoupling of coordination from execution.
Written by urgent.news from InfoQ's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
