Lyft Moves Streaming Fleet to Apache Flink Kubernetes Operator

Lyft has transitioned its vast Apache Flink streaming fleet from a custom Kubernetes operator to the Apache Flink Kubernetes Operator. In a post on 31 August, engineers Maheep Myneni, Arda Kuyumcu, and Prem Santosh Udaya Shankar highlighted the benefits of this move, including seamless last-state upgrades, automated scaling, and resource autotuning.

The in-house operator was developed in 2020 as the open-source community had yet to build a dedicated control plane for Flink on Kubernetes. The legacy operator required manual intervention for dual-deployment upgrades, lacked robust savepoint triggers, and had limited memory reservation options. To address these issues, Lyft adopted the Apache operator, which treats last-state upgrades as a first-class feature and restores from high-availability metadata or the latest checkpoint even if the JobManager is unhealthy.

The Apache operator simplifies the deployment process, eliminates the need for rewriting Jsonnet templates, and starts JobManagers to manage TaskManager lifecycles. Moving off the legacy operator also replaced dual deployments with stop-then-start deploys, reducing downtime from 3 to 6 minutes to about 20 minutes for larger jobs.

Lyft also adopted FlinkBlueGreenDeployment, a custom CRD that allows new versions to run alongside old ones before cutover. Upgrading to Flink 1.19 enabled in-place scaling and unlocked the KinesisStreamsSource, providing autoscaling with record backlog metrics for Kinesis streams. Lyft split features by criticality, using in-place autoscaling without autotuning for pricing and routing jobs, while less critical workloads accepted restarts for resource tuning. The autoscaler has significantly reduced overprovisioning, saving a few million dollars annually.