One instance was bad at its job and nothing was designed to notice
Checkout errors sat between three and eight percent for most of a Thursday. Not enough to page, because our threshold is one percent sustained across the service and the service was at 0.4. Enough that retries covered most of it and support saw a thin, steady trickle of people who had tried twice and given up. It took three hours to establish that one instance out of twenty-four was producing…
On a Thursday, checkout errors ranged from three to eight percent. The threshold for triggering an alert was one percent sustained across the service, which was at 0.4 percent. Despite this, retries resolved most of the issue, and support observed a steady trickle of users who had attempted to retry twice and given up. After three hours, it was determined that one instance out of twenty-four was responsible for nearly all of the errors.
The host had a degraded network interface, causing roughly one out of every ten connections to that instance to stall and eventually time out. Despite appearing ordinary, the instance was serving traffic, using CPU, reporting metrics, and responding to the load balancer's health check without any issues.
Two assumptions proved to be the root cause of the problem. The health check assessed the instance's performance independently of actual requests, and a system with a faulty network interface could still report that it was functioning normally. Dashboard data, which aggregated across the entire fleet, masked the impact of the problematic instance's eight percent error rate as a rounding error.
To address the issue, structural changes were implemented. Per-instance error rates and latency were added to the service dashboard, with an alert triggering when any single instance's performance exceeded five times the fleet median for five minutes. This comparison-based alert required no tuning and was effective at any traffic level.
The load balancer also incorporated passive health checking, which ejected targets based on real request failures rather than synthetic success checks. Additionally, hardware health events from the provider were integrated with other notifications, arriving alongside them in the same channel.
The most significant change involved adjusting the failure model. Instead of merely categorizing instances as either "working" or "gone," the new model provided intermediate states, offering a more nuanced understanding of the situation. This shift in perspective allowed teams to focus on investigating suspected instances before removal, rather than spending valuable time debating whether a problematic machine needed to be replaced.
Since that Thursday, most of the challenges faced since have fallen within this improved failure model, ranging from three to eight percent errors.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
