Case study (hypothetical): the provider disabled our webhook endpoint after a weekend of 500s, and nobody noticed for 9 days
Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run This is a hypothetical case study, assembled from a pattern that turns up again and again with webhook integrations. The company, timeline and numbers are invented. The setup A company runs an online booking platform. A third-party payments provider sends it webhooks for completed payments, refunds…
This hypothetical case study explores a provider disabling a webhook endpoint after a weekend of failed requests, leading to unnoticed downtime for nine days. The online booking platform received stable webhook notifications for completed payments, refunds, and disputes for two years. The handler verified signatures, stored events, returned a 200 status, and processed payments in a background job.
However, after a routine deploy on Friday evening, the webhook route received parsed JSON instead of raw data, causing signature verification errors and resulting in 500 errors. The error monitoring tool initially classified these errors as someone probing the endpoint with fake requests and snoozed the issue. On Monday, the team found that customers were still facing pending payments, and after rolling back the middleware change, they confirmed that test webhooks from the provider's dashboard succeeded.
The provider's rules stated that an endpoint failing continuously for more than a certain period would be automatically disabled, and the technical contact, who had left the company eight months earlier, was not notified due to an outdated email address. As a result, the webhook endpoint was disabled for about 62 hours, and the platform received no webhooks, leading to a significant discrepancy between the number of paid bookings in the platform and the number of payments reported by the provider.
The solution included re-enabling the endpoint, replaying missed events, monitoring the provider's endpoint status, fixing contact details, changing the response status for verification failures, and implementing a proper reconciliation job for real-time tracking of payments and bookings.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.