Your Pods Are Not Draining: I Checked

Multiple studies on Kubernetes pods reveal different outcomes when pods are terminated. A Go tool named drainwatch was built to measure the effects of pod termination on TCP and UDP connections. Drainwatch runs trials with five termination scenarios, five repeats each, and measures the time it takes for connections to end and the events that trigger them.

The study found that the application's behavior upon receiving SIGTERM significantly influences the outcome for TCP connections. Three applications were tested: one exiting immediately, one draining, and one ignoring SIGTERM. The immediate exit application terminated connections within 45 milliseconds, while the draining application maintained all connections until its 25-second window.

Ignoring SIGTERM kept connections alive until the kubelet intervened, which occurred within 62 milliseconds of the 30-second grace period. The kubelet, not the application, decides when to apply SIGKILL. An unexpected finding was that UDP connections behave differently during pod termination. Unlike TCP, UDP connections are connectionless, and their fate depends on whether a replacement pod is available.

If a replacement pod exists, UDP flows are silently migrated to the new pod, while without a replacement pod, UDP connections are rejected outright with a 177-millisecond median delay. These findings reveal important insights into Kubernetes pod termination behavior, emphasizing the need for application-level handling and detailed understanding of connection types (TCP and UDP) during pod termination.