Webhooks arrive out of order. Should your handler reorder them, or design so order doesn't matter? Let's argue about it
Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run Every webhook provider's documentation has a sentence like this somewhere: "We don't guarantee delivery of events in the order in which they were generated." Most teams read it, nod, and build a handler that assumes order anyway. It works for months, because most of the time events do arrive in…
When webhook providers don't guarantee delivery of events in the order they were generated, teams often debate whether to reorder events or design handlers that don't depend on order. Both approaches have their merits and drawbacks.
Camp 1 advocates buffering incoming events, sorting them by a reliable timestamp or sequence number, and processing them in order. This approach keeps business logic simple and matches the provider's state machine. It makes audit trails straightforward, as events appear in a chronological history. However, timestamp sorting isn't foolproof, as events with identical timestamps or out-of-order clocks can lead to misordering.
Buffering introduces stateful components, requiring storage, locking mechanisms, and its own failure handling. Retries can delay events, making it impractical to hold events for extended periods.
Camp 2 proposes treating each webhook as a notification of state changes and making the handler order-independent. The two common techniques are fetching the current state from the provider's API after each event and comparing it to the stored version, or using version, sequence number, or updated_at fields to prioritize events.
This method ensures correct state convergence regardless of event timing, handles duplicates, and doesn't rely on buffering. However, it incurs extra API calls, potentially causing latency and rate limit issues. It also loses intermediate states, meaning events like "subscription becomes active" might be missed.
In practice, a hybrid approach is often adopted. For stateful objects like subscriptions, customers, or orders, handlers should aim to process events in order by relying on version or updated_at fields and fetching the latest state. This ensures data consistency and enables accurate side effects, such as sending welcome emails when a subscription becomes active.
For non-state-changing events, like payment attempts or email bounces, handlers should process each event once, idempotently by event ID, and don't depend on event order. This approach handles missed events and ensures consistent processing even if events arrive out of sequence.
To handle deletes, tombstones or "deleted at" markers should be implemented to prevent overwritten updates from late arriving events. For objects spanning multiple providers, there's no single version number, so a different approach, like using unique identifiers, may be necessary.
When evaluating webhook handlers, consider testing scenarios where events arrive out of order, are duplicated, or go missing. If any answer involves the data being incorrect until manually corrected, it indicates a need for a more robust solution.
Ultimately, webhook handlers should strive to trust the provider's latest state rather than relying solely on payload data. Fetching current state after processing an event and scheduling follow-up tasks, like sending notifications, can ensure accurate and timely actions. This approach minimizes the risks associated with out-of-order delivery and provides a more reliable webhook handling experience.
Rewritten by urgent.news; may contain errors — check the original.