Taming Kafka Lag Spikes with KEDA Scale-to-Zero
How we turned always-on Kafka sinks into on-demand workers that shrug off nightly bombardments — by scaling on the right signal, tuning per-pod drain rate, and keeping autoscaling from sabotaging itself. Every number in this post is measured from a local lab you can run yourself — the full code is on GitHub , and the Appendix has the commands. The problem We run a fleet of Kafka sinks — consumer…
The challenge lies in balancing two seemingly opposing goals for Kafka sinks: minimizing idle costs during quiet periods and rapidly absorbing spikes without lag accumulation. Traditional Kubernetes HPA based on CPU or memory metrics fails because sink work is I/O-bound, waiting on Kafka polls and database writes rather than heavy CPU usage. Consumer lag therefore becomes the key signal to scale on, with KEDA providing the ability to scale to zero replicas when idle.
Three key moves address the problem:
1) Use KEDA to scale Kafka sinks based on consumer group lag, setting a trigger to scale from zero as soon as lag exceeds zero. This directly measures "work waiting to be processed". A max replica count of 12 (equal to the partition count) caps the scale.
2) Increase each pod's drain rate by making consumers fetch larger batches per poll and write batches atomically. Instead of one upsert per row, the tuned consumer processes batches of ~350 records each with max.poll.records set to 350. This reduces the number of round-trips per record and speeds up backpressure propagation, allowing KEDA to scale down quicker after the spike.
3) Combine the lag-driven autoscaling with the tuned pods to achieve the "always-on to on-demand" shift. During idle periods, KEDA scales all the way to zero pods, incurring no cost. When a burst arrives, lag triggers rapid scaling to a full fleet of pods, which drain the backlog efficiently and then scale back to zero.
Measured results show a one million-message burst being processed by 12 pods in roughly 96 seconds after the burst ends, versus a naive implementation that would have taken about 1.5 hours. Idle resource usage drops to zero replicas, achieving the cost savings goal while ensuring spikes are handled quickly and lag stays under control. The solution uses KEDA to scale Kafka sinks based on consumer lag, paired with tuned consumer configurations to maximize drain rate per pod.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
