Testing webhook retries deterministically: a fault sequence per Idempotency-Key

In a recent scenario, a webhook receiver was designed to be slow and unreliable on purpose. A reader's comment led to a significant realization: a 10% error rate indicates sender retries, but it doesn't confirm whether those retries are executed correctly and cannot be verified through continuous integration (CI). The real issue at hand is that the service sends a webhook, the receiver processes it, and before the service reads a 200 response, the connection is lost.

Upon retrying with the same idempotency key, the receiver either creates a second order or recognizes the key and responds with "already done." To address this, a new mode was introduced, allowing a list of steps to be walked per key. The endpoint accepts a JSON list of steps, and for each distinct idempotency key, the steps are followed in order.

The first call with a key executes step one, the second call with the same key executes step two, and a new key restarts from step one. The three actions available are 'respond', 'drop', and 'timeout'. The 'respond' action sends a status, body, and optional delay, while 'drop' logs the request and cuts the connection without delivering a body.

The 'timeout' action holds the connection for up to 30 seconds before cutting it. After the last step, 'on_end' determines the next action, with the default being to repeat the last step indefinitely. Other considerations include hashing the key before storage, the limitations on header names, and the handling of responses. The endpoints run on Cloudflare Workers, with a free plan offering 3 endpoints, 30 requests per minute, and 500 requests per day, with a limit of 20 steps per sequence.