The valley of webhooks

The "valley of webhooks" describes the challenges faced when integrating with third-party services through webhooks. These integrations are built multiple times across different companies for various providers, and they always follow a similar path: the truth about customer data lives in someone else's database. Users typically reside in identity providers, subscriptions in Stripe, bounces in email services, and the product needs this data locally. To achieve this, webhooks are subscribed to and a copy of the data is kept.

The initial setup of these systems is often thought to be a simple task, with just one route that parses JSON and updates a row. However, this quickly evolves into a much larger project. The system must include signature verification to prevent tampering, a deduplication table to handle duplicate deliveries, a buffer to manage the "at-least-once" nature of webhook deliveries, a bootstrapping mechanism to handle events that occur before the webhook delivery, and a reconciliation cron job to periodically compare provider lists with the local database.

While building these systems, the reporter learned that they never truly trust their copies and instead re-derive the data from scratch every night. This is a confession that the copy is not reliable and that there's no way to know when it's wrong. The reporters emphasizes the lack of trust due to the drift between the provider's data and the local database, which can go unnoticed.

An example mentioned is a customer who cancelled months ago, but the database still showed them as active, due to a missed deletion event between Stripe and the local system.

Debugging these issues is also a challenge, as each provider has its own dashboard for webhook configuration, and the documentation is inconsistent. Debugging requires comparing different logs and checking multiple sources, making the process cumbersome. The reporter acknowledges that every integration is an attempt to rebuild an ordered log of events, yet every provider provides only a stream of notifications. Each consumer must independently reassemble this history, leading to potential discrepancies and inconsistencies.

The ultimate problem with webhooks is their inherent limitations. They are notifications of events happening, not a reliable method of transferring datasets. This is evident in the lack of ordering, completeness, bootstrap functionality, and verifiability in webhook systems. These are all crucial aspects when it comes to replicating data between systems.

The reporter argues that webhooks were designed to trigger side effects, not to transfer complete datasets. The prevalence of webhooks as a cheap solution (just an HTTP POST) and their widespread adoption by providers has led to a situation where many consumers are working around the limitations of the webhook system, resulting in a valley of workarounds and inefficiencies.

Throughout the evolution of webhooks, there has been no conscious decision to adopt them for data replication purposes. The term "webhook" was coined in 2007, and early uses were indeed suitable for triggering side effects. However, as the use cases expanded, it became clear that webhooks were not the ideal tool for replicating data.

The valley of webhooks represents a local optimum, where the temporary benefits of using webhooks outweigh the long-term drawbacks. Crossing over to a higher peak, where more robust data replication methods are employed, requires navigating through various compromises and workarounds.

In conclusion, the "valley of webhooks" highlights the challenges and complexities of integrating with third-party services using webhooks. It emphasizes the need to recognize the limitations of webhooks when it comes to replicating data and to seek more reliable and comprehensive solutions for data transfer between systems.