Who Kills Your Background Worker, and How Soon? A Shutdown Test Plan

Teams typically focus their testing efforts on the initial moments of a process, but pay little attention to the latter stages. This imbalance can be costly because if a worker starts up correctly but is terminated while performing a write, it may result in incomplete batch processing, duplicate side effects, or lingering queue items.

The following plan aims to evaluate the shutdown behavior of your existing system: which signal is sent, how long the platform waits before taking action, and what remains if the process is terminated prematurely. Please note that the details provided are for testing purposes, and the figures you obtain will be specific to your environment.

When a system halts a process, it typically goes through two stages: a polite request (SIGTERM) followed by an irrevocable signal (SIGKILL) after a set period. The duration of this interval is determined by platform policy and often does not align with the expectation. The man page for signal(7) offers detailed information on the underlying kernel-level mechanisms.

By default, SIGTERM causes a process to terminate, so assigning a handler converts it into a request that can be responded to. On the other hand, SIGKILL cannot be intercepted or delayed; any unsaved work before it arrives is lost. If your process operates as PID 1 within a container, certain signals with default dispositions may not effectively bring it to an end, so it's essential to confirm this behavior for your specific runtime rather than relying on assumptions.

In a bash environment, an exit status of 143 generally indicates that the default action (128 + 15) has been executed, signaling that the handler was not invoked. To accurately assess your system's performance, capture three timestamps for each run: the moment the handler detects the signal, the time when the final durable write is acknowledged, and the exact moment the process concludes.

Comparing these timestamps will allow you to assess the available time budget instead of relying on estimations. The outcome of this analysis can be represented by a simple shutdown-budget harness. The prerequisites for this harness are straightforward: Python 3, a POSIX shell, and a worker that utilizes a queue with unique item identifiers.

The provided Python script serves as a template for monitoring the shutdown process. It sets a budget for the shutdown period and a margin for network acknowledgments and data flushing. The script defines functions to handle the SIGTERM signal and emits events at specific timestamps. By adjusting the environment variables SHUTDOWN_BUDGET_S and SHUTDOWN_MARGIN_S, you can tailor the test scenario to your specific requirements.

To measure the shutdown behavior locally, a Bash script is provided as an example. This script runs the worker process, simulates a SIGTERM signal, and measures the elapsed time from signal reception to process termination. The results are then displayed using awk. It is crucial to run the test twice: once with the signal handler enabled and once with it disabled to validate that the handler is correctly handling the SIGTERM signal.

The differences between the two sets of results will indicate the effectiveness of your signal handling mechanism.

The test scenarios provided offer a range of conditions to cover different aspects of the shutdown process. The first scenario involves a graceful termination using SIGTERM in the middle of a batch, allowing you to observe the behavior when the handler is activated. The second scenario tests a hard stop using SIGKILL in the middle of the same batch, providing insight into the system's reaction to an unhandled termination signal.

Comparing the outcomes of these scenarios will help you understand how your process behaves under various termination conditions and whether your shutdown budget is being appropriately managed.