Urgent.News

What's breaking now, across thousands of outlets.

Tech

Case study (hypothetical): the provider disabled our webhook endpoint after a weekend of 500s, and nobody noticed for 9 days

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run This is a hypothetical case study, assembled from a pattern that turns up again and again with webhook integrations. The company, timeline and numbers are invented. The setup A company runs an online booking platform. A third-party payments provider sends it webhooks for completed payments, refunds…

This hypothetical case study explores a provider disabling a webhook endpoint after a weekend of failed requests, leading to unnoticed downtime for nine days. The online booking platform received stable webhook notifications for completed payments, refunds, and disputes for two years. The handler verified signatures, stored events, returned a 200 status, and processed payments in a background job.

However, after a routine deploy on Friday evening, the webhook route received parsed JSON instead of raw data, causing signature verification errors and resulting in 500 errors. The error monitoring tool initially classified these errors as someone probing the endpoint with fake requests and snoozed the issue. On Monday, the team found that customers were still facing pending payments, and after rolling back the middleware change, they confirmed that test webhooks from the provider's dashboard succeeded.

The provider's rules stated that an endpoint failing continuously for more than a certain period would be automatically disabled, and the technical contact, who had left the company eight months earlier, was not notified due to an outdated email address. As a result, the webhook endpoint was disabled for about 62 hours, and the platform received no webhooks, leading to a significant discrepancy between the number of paid bookings in the platform and the number of payments reported by the provider.

The solution included re-enabling the endpoint, replaying missed events, monitoring the provider's endpoint status, fixing contact details, changing the response status for verification failures, and implementing a proper reconciliation job for real-time tracking of payments and bookings.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

A one-sample t statistic in Python: what the number actually compares

A two-point difference can be a big deal or ordinary sampling noise. The t statistic compares that difference with the estimated uncertainty in the sample mean.

  • One-sample t statistic compares sample mean to hypothesized value
  • Larger t value with smaller p-value indicates stronger evidence
  • Assumes independent normal observations for valid results

How to fix 'too many missed start times' in Kubernetes CronJobs (and why your job silently stopped running)

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run You check on a Kubernetes CronJob that should run every minute and find that it hasn't created a Job…

  • CronJobs stop running after 100 missed start times in default Kubernetes
  • Missed start times can occur due to suspension, controller issues, clock skew, or frequent schedules
  • Set startingDeadlineSeconds to allow runs within a specified late time window

Private Image Originals and Thumbnails: Retention, Resizing, and Signed Downloads

Short answer: use private object storage for originals and generated thumbnails, run resizing in your Node.js worker or an image service, and return short-lived signed download links.

  • Separate private originals and generated thumbnails in distinct prefixes (origins/, thumbs/)
  • Use unique keys for variants (originals/{tenantId}/{imageId}/source.jpg) and record them in DB
  • Generate signed GET URLs for downloads, log request ID, image ID, variant, attempt, and time

More from Wednesday 18 November →