Designing a Webhook Delivery System for 10 Million Events a Day

Hello, I'm Maneshwar, and I'm building LiveReview - an AI code review tool designed to be business-critical for your systems. I invite you to join us, help developers discover the project, give it a try, and share your feedback to improve the product. Someone on your team will inevitably mention "webhooks" during a planning meeting or when looking at a Jira ticket with a minimal description.

They're correct about webhooks being a simple POST request, but this simplicity can lead to serious issues when handling large volumes of events.

In this story, I'll describe the journey of designing a webhook delivery system capable of processing 10 million events daily. The initial approach was straightforward: when an event occurs, post it to the customer's URL and wait for a 200 OK response. However, when this system was deployed in production, it revealed significant problems.

During peak times, the customer's endpoint might take several seconds to respond, causing the request handler to wait on their infrastructure. Latency graphs spiked, connection pools drained, and users experienced timeouts. Worse still, the system failed silently, as events were lost without any trace. The bug wasn't "it was slow," but rather the system's availability was dependent on the customer's availability, located in the hot path.

To address this, I introduced the transactional outbox pattern. Instead of posting events immediately, the handler writes them to a table in the same transaction as the business change that caused them. A separate worker then polls for pending rows and performs the actual posting. This ensures that events are durable and can be retried if delivery fails. However, this approach introduces head-of-line blocking, where one customer's slow endpoint can block the progress of all other events.

To mitigate this issue, I implemented per-tenant concurrency limiting. A dispatcher manages the flow of events, ensuring each customer has a fair share of the worker pool. This dispatcher keeps track of how many workers are currently busy with each customer and prioritizes events accordingly. By doing so, the performance of a slow customer no longer affects the overall system, as their issues are isolated to themselves.

This design also allows for weighted fairness, ensuring that enterprise tier customers aren't starved by free-tier customers emitting a high volume of events.