Urgent.News

What's breaking now, across thousands of outlets.

Tech

I made the database fail on purpose in a legacy checkout: /health stayed 200 and 3 of 214 specs turned red

The complaint arrived as a screenshot: a spinner, then an error page, on a Tuesday afternoon. Three customers could not pay. By the time I opened the dashboard for that service, everything on it was green. Four instances up, CPU under 20%, uptime for the month a shade over 99.9%, no alert fired. The /health endpoint had answered 200 the whole time. Here is that endpoint, with the variable names…

A reporter investigated a database failure in a legacy checkout system. The support team sent a screenshot showing a spinner, then an error page, on a Tuesday afternoon. Three customers were unable to complete payments. Despite all system metrics appearing normal, the /health endpoint remained at 200, while 3 out of 214 tests turned red.

The service, nine years old and written in Node and Express, had 40k lines of code and an 87% test coverage. The team questioned why the failure went unnoticed, but the reporter found a solution using deterministic simulation testing. By injecting a fault at a specific query, the same failure could be reproduced consistently, unlike previous attempts with random packet drops and connection kills.

The reporter added two checks to the system: a health check that monitors dependencies and a seeded fault injector around the database pool. By running the test suite with these additions, they were able to identify the specific failure and its impact on the payment flow. The dashboard remained green, but the reporter's investigation revealed a previously undetected issue in the system.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

Private Image Originals and Thumbnails: Retention, Resizing, and Signed Downloads

Short answer: use private object storage for originals and generated thumbnails, run resizing in your Node.js worker or an image service, and return short-lived signed download links.

  • Separate private originals and generated thumbnails in distinct prefixes (origins/, thumbs/)
  • Use unique keys for variants (originals/{tenantId}/{imageId}/source.jpg) and record them in DB
  • Generate signed GET URLs for downloads, log request ID, image ID, variant, attempt, and time

How to fix 'too many missed start times' in Kubernetes CronJobs (and why your job silently stopped running)

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run You check on a Kubernetes CronJob that should run every minute and find that it hasn't created a Job…

  • CronJobs stop running after 100 missed start times in default Kubernetes
  • Missed start times can occur due to suspension, controller issues, clock skew, or frequent schedules
  • Set startingDeadlineSeconds to allow runs within a specified late time window

Case study (hypothetical): the provider disabled our webhook endpoint after a weekend of 500s, and nobody noticed for 9 days

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run This is a hypothetical case study, assembled from a pattern that turns up again and again with…

  • Provider disabled webhook endpoint after 9 days of 500 errors
  • Online booking platform experienced unnoticed 9-day downtime
  • Technical contact outdated email address led to disabled endpoint

How to fix 'Blocked due to access forbidden (403)' in Google Search Console (it's usually your WAF or bot rules)

Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com Search Console's Pages report has a reason that looks simple: Blocked due to access forbidden (403) .

  • Confirm issue by selecting URLs and performing URL Inspection and live URL test
  • Identify WAF or bot rules layer causing 403 response through headers and body analysis
  • Verify fix by ensuring Googlebot requests return 200 status in live test

Should every URL end with a trailing slash? Let's argue about it

Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com Few URL questions generate as much quiet frustration as this one.

  • Trailing slash debate centers on URL format preference
  • Supporters argue alignment with static hosting platforms
  • Opponents prefer URLs without trailing slashes for ease of use

My analytics undercounted my audience by an order of magnitude

Last week my own funnel report told me something comfortable and wrong: 123 human visitors in 30 days . At that number, the only rational conclusion is "I have no audience", and the rational move…

  • Analytics undercounted audience by an order of magnitude
  • Bot visits and unclassified visits omitted from human count
  • New metrics report denominator, change date, and known-unknown bucket

More from Wednesday 18 November →