48 Hours Chasing a Duplicate Nightly Job: SIGTERM Was Landing on the Wrapper

Two consumers were simultaneously processing the same queue queue, without being aware of each other's existence. The nightly job typically took eleven minutes to complete, but when metrics revealed the same batch was processed twice, the reporter initially thought it was due to duplicate messages. It was only after 48 hours that the reporter realized the deploy script was terminating the wrong process for months.

The primary issue was that the job was double-processing after every deployment, always appearing within fifteen minutes of a deployment. The only clue pointing to the shutdown path was the timing of the double processing, rather than the message broker.

Upon investigating, the reporter discovered that the worker count on the host showed two PIDs, while the supervisor reported zero. The sequence of events was as follows: the deploy script read a pidfile and sent a SIGTERM to the recorded PID. The supervisor marked the service as stopped and initiated a new instance. However, the old Python process remained alive, reparented and still holding a queue connection.

The reporter initially suspected a missing signal handler and added one, logging every delivery. However, this did not resolve the issue. The second theory was a stuck network read, so the reporter added timeouts everywhere and eventually used os._exit. Still, two processes remained.

The reporter's mistakes included reading the pidfile as ground truth, as the wrapper script wrote echo $$ worker.pid before starting Python. The recorded PID belonged to the shell, not the consumer. Killing the shell left the child orphaned, reparented to init, and perfectly healthy. The reporter also overlooked the importance of checking process group and session IDs to determine if a SIGTERM would be a safe broadcast or a disaster.

To better understand the process behavior, the reporter built a three-file harness that reproduced the failure in about five seconds. The worker recorded every event as JSON lines, including the PID, PPID, and process group at each event. By running this harness, the reporter could determine if a SIGTERM was being handled or ignored.

The reporter's reproducible test involved killing the wrapper and observing the surviving processes. If no Python processes were found, it indicated that the signal handler never ran, confirming that the supervisor was not correctly terminating the process.