Netflix Adopts Cloud-Native Job Queueing System Kueue to Replace an In-House Solution

Netflix has transitioned most of its batch workloads to Kueue, an open-source cloud-native batch job execution system, in place of its previous in-house solution. The company mapped the capabilities of its existing system to Kueue's functionality and benefited from new features that would have been expensive to implement within its custom solution. Engineers utilized API parity with the current system to minimize risks and facilitate a smooth migration process.

Netflix, over the years, developed a series of bespoke solutions for Titus, its container platform, including Compute Managed Batch (CMB) for managing and executing batch jobs on containerized compute. However, many of CMB's key features have since been incorporated into open-source projects within the Kubernetes ecosystem, and developing new features has become increasingly difficult due to the solution's lack of tight integration with Kubernetes.

Netflix decided to replace CMB with Kueue, given its extensive capabilities, flexibility, widespread adoption, and rapid innovation. As part of the migration, the team responsible ensured a seamless transition for existing CMB users and confirmed Kueue could meet the required container launch rate and maximum throughput. The migration process was tenant-bound, enabling easy rollback in case of any issues.

Netflix currently manages millions of batch workloads in production using Kueue, and the migration is still ongoing. Engineers have also improved average resource utilization by leveraging preemption-based fair sharing, enabling idle capacity to be lent to other tenants while maintaining reservation semantics. The team shared several lessons learned, including the importance of migrating the most complex use case early on and running load tests in a non-prod environment to fine-tune performance-related configuration options.