{
  "id": 5136177,
  "title": "Designing a Webhook Delivery System for 10 Million Events a Day",
  "url": "https://urgent.news/2026/09/02/designing-a-webhook-delivery-system-for-10-million-events-a-day",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-02T17:30:40.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/lovestaco/designing-a-webhook-delivery-system-for-10-million-events-a-day-2p5d"
  },
  "original_language": "en",
  "account": "Hello, I'm Maneshwar, and I'm building LiveReview - an AI code review tool designed to be business-critical for your systems. I invite you to join us, help developers discover the project, give it a try, and share your feedback to improve the product. Someone on your team will inevitably mention \"webhooks\" during a planning meeting or when looking at a Jira ticket with a minimal description. They're correct about webhooks being a simple POST request, but this simplicity can lead to serious issues when handling large volumes of events.\n\nIn this story, I'll describe the journey of designing a webhook delivery system capable of processing 10 million events daily. The initial approach was straightforward: when an event occurs, post it to the customer's URL and wait for a 200 OK response. However, when this system was deployed in production, it revealed significant problems.\n\nDuring peak times, the customer's endpoint might take several seconds to respond, causing the request handler to wait on their infrastructure. Latency graphs spiked, connection pools drained, and users experienced timeouts. Worse still, the system failed silently, as events were lost without any trace. The bug wasn't \"it was slow,\" but rather the system's availability was dependent on the customer's availability, located in the hot path.\n\nTo address this, I introduced the transactional outbox pattern. Instead of posting events immediately, the handler writes them to a table in the same transaction as the business change that caused them. A separate worker then polls for pending rows and performs the actual posting. This ensures that events are durable and can be retried if delivery fails. However, this approach introduces head-of-line blocking, where one customer's slow endpoint can block the progress of all other events.\n\nTo mitigate this issue, I implemented per-tenant concurrency limiting. A dispatcher manages the flow of events, ensuring each customer has a fair share of the worker pool. This dispatcher keeps track of how many workers are currently busy with each customer and prioritizes events accordingly. By doing so, the performance of a slow customer no longer affects the overall system, as their issues are isolated to themselves. This design also allows for weighted fairness, ensuring that enterprise tier customers aren't starved by free-tier customers emitting a high volume of events.",
  "summary": "Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. Somebody on your team is going to say it, probably in a planning meeting, probably while looking at a Jira ticket that has three words in it. \"Webhooks?…",
  "key_points": [
    "Designed webhook delivery system for 10 million events daily",
    "Implemented transactional outbox pattern for durable event storage",
    "Introduced per-tenant concurrency limiting to prevent head-of-line blocking"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}