What is a webhook? How webhooks work, and the six ways they break in production

A webhook is an HTTP POST that one system sends to a registered URL when an event occurs. The receiver registers once, and the sender calls the webhook every time the event happens. An example of a signed webhook request includes headers like webhook-id, webhook-timestamp, and webhook-signature, along with a JSON body containing event details.

Webhooks offer several advantages over polling, including fewer requests, better performance, and more efficient resource usage. However, they can also fail in production due to various reasons.

There are six main ways webhooks can break in production:

1. Anatomy of a signed request: The signature verification process can fail if the raw body computed by the HMAC does not match the exact bytes received. Even minor manipulations to the raw body, such as whitespace changes or trailing zeros, can cause verification to fail. Additionally, some frameworks may already parse the JSON body before the handler runs, resulting in an incorrect body being handed to the handler and causing the signature verification to fail.

2. Polling vs push: Webhooks differ from polling in that the sender initiates the request when an event occurs, whereas polling relies on the receiver periodically checking the sender's API. This difference in request initiation has implications for how failures are handled and the overall system reliability.

3. Retries and duplicates: Webhook delivery may result in duplicate deliveries if the sender retries failed requests. This can lead to issues such as double loyalty points or double receipts being sent to the customer. Moreover, the sender's decision on how to handle failures, such as retries or timeouts, can impact the overall behavior of the webhook integration.

4. Out-of-order delivery: Webhooks can be delivered out of order, which may cause inconsistencies in the receiver's state or actions. Proper handling of out-of-order deliveries is crucial to ensure the system maintains a consistent state.

5. Secret rotation: Webhook signatures rely on a shared secret between the sender and receiver. Periodically rotating this secret is essential for maintaining security. However, if not handled correctly, secret rotation can lead to disruptions in the webhook communication, as receivers may need to update their verification mechanisms accordingly.

6. Out-of-order handling: The receiver must be able to handle out-of-order webhook deliveries, ensuring that the system maintains a consistent state despite the potential for request order to be disrupted. Failing to handle out-of-order deliveries correctly can lead to inconsistencies in the system's behavior and potentially cause errors or loss of data.