How do you actually test for a failure that leaves no trace?

A specific kind of bug was being investigated that is difficult to explain to those who haven't encountered it. This bug presents as a test suite that passes, a system that appears to work correctly, and a business event that is missing entirely. There are no error logs, as the system's point of view is that nothing happened. The reporter wanted to determine if the designed pattern was genuinely safe or merely seeming so. They set out to intentionally break this pattern.

The reporter created two variants of a minimal webhook handler and introduced a controlled failure using a SIGKILL signal sent to the process at a specific point after a response is sent or a database write is completed. This simulates a failure scenario such as a host reboot or an out-of-memory kill. The same event ID was resent after the process was restarted, mimicking how a provider behaves when it doesn't receive a response.

In one trial, the "ack-then-persist" variant lost the event every single time. The "200" was sent out, but the process died before the database insert could complete, resulting in no retry to save the event. The provider received a successful delivery and had no reason to resend the event. On the other hand, the "persist-then-ack" variant did not lose any events, indicating that the persistence operation completed before the acknowledgment.

The reporter expected to observe a race condition between checking for an existing event and attempting to insert it, but the actual problem was different. Despite both mechanisms resulting in no duplicate rows, the check-then-insert mechanism faced a race condition where two goroutines both perceived no existing row and attempted to insert it. One succeeded, while the other encountered a unique constraint error.

In conclusion, the reporter emphasized that it's crucial to persist data before acknowledging its receipt. A unique constraint on the deduplication key within the database schema is a more reliable solution than relying solely on application-level logic to prevent duplicate rows. Additionally, an atomic INSERT operation combined with ON CONFLICT DO NOTHING is preferable over check-then-insert, as it provides a cleaner response to the sender and avoids unnecessary retries due to errors.

The test highlighted the importance of real-time data integrity and the potential pitfalls of relying solely on application-level logic for handling concurrent operations.