Webhook Security Is Distributed Systems Security

Webhook security is a critical aspect of distributed systems security. A webhook is a message from a queue you do not own, arriving at an address anyone on the internet can find. It is not just an HTTP request, but a message with no delivery guarantees, arriving unordered and retried. Most webhook bugs are distributed-systems failures disguised as HTTP issues. Replay, duplicate delivery, reordering, and retry storms are common problems.

The trust boundary lies in the signature, which separates a real event from a forged one. The signature check is the whole wall, not just a formality. Developers often verify the wrong bytes, leading to verification failures on legitimate events. The solution is to verify the raw body before any parsing occurs.

While using a signature provides origin and integrity, it does not guarantee freshness. To address this, include a timestamp within the signed payload and set a window for acceptable timestamps. This closes the replay hole, as a replayed event with a valid signature and timestamp will still be accepted.

At-least-once delivery means duplicates are possible, leading to duplicate charges and shipping. To prevent this, ensure the handler is idempotent by using an atomic operation to check and claim the event ID, storing it atomically. This prevents processing the same event multiple times.

Lastly, handle the unordered delivery of events by including a version or monotonic sequence in the event. Ignore events older than the current state, ensuring the handler is not dependent on arrival order. This approach addresses the distributed-systems challenges inherent in webhook security.