{
  "id": 20281227,
  "title": "Your Retry Logic Might Be the Reason Your System Won't Recover",
  "url": "https://urgent.news/2026/11/16/your-retry-logic-might-be-the-reason-your-system-wont-recover",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-16T08:57:14.000Z",
  "source": {
    "name": "HackerNoon",
    "slug": "hackernoon",
    "url": "https://hackernoon.com/your-retry-logic-might-be-the-reason-your-system-wont-recover?source=rss"
  },
  "original_language": "en",
  "account": "The phrase \"don't trouble the trouble\" highlights the importance of not exacerbating the issue when dealing with distributed systems. This wisdom extends to the realm of software engineering, where engineers typically follow a common practice: when a call fails, they attempt it again. Most frameworks silently implement a sensible default of three attempts without any additional configuration. For a single client, this approach is effective, as network issues or mid-flight instance recycles can be resolved through retrying. However, problems arise when thousands of clients independently adopt the same instinct at the same time.\n\nConsider a service that can handle 1,000 requests per second with ample headroom. If something downstream degrades, such as a database failover, an evicted cache node, or a dependency with latency exceeding its timeout, response times slow down, and calls hit the one-second timeout. Since every caller is configured to retry twice, the service is now presented with 3,000 requests per second. Its capacity remains at 1,200 requests per second. Consequently, every request times out, triggering additional retries. The issue is no longer connected to the initial fault but becomes self-sustaining, leading to a queue collapse as the load amplifies.\n\nThe root cause of this self-sustaining queue collapse is that the original fault—slow dependency—no longer generates the traffic. Instead, the load is self-sustaining, driven entirely by the service's response to the fault. Healing the dependency does not restore normal operation, as the retry traffic has taken over. This scenario mirrors the historical phenomenon of congestion collapse observed by network engineers, which recurs every time a new layer of the stack introduces a default retry policy.\n\nOne key challenge is that a retry decision is made with insufficient information. When a caller times out, it knows only that no response arrived within the deadline. It does not know whether the request was lost in transit, received and dropped, or still executing. A timeout is merely the absence of a signal, not a definitive failure message. Consequently, clients make assumptions, typically assuming the issue is transient and retrying accordingly. These retries are most aggressive when the system is already overloaded, leading to even more detrimental amplification.\n\nTo mitigate this issue, several strategies can be employed. Firstly, caps can be placed on retries as a fraction of total traffic rather than per request. For example, instead of allowing each request to be attempted three times, the rule could be that retries may not exceed 10% of total requests. Under normal conditions, this does not alter the system's behavior. However, during failures, the amplification is bounded by construction. Additionally, implementing exponential backoff with jitter is crucial. Without jitter, every client failing simultaneously would synchronize their retries, resulting in a synchronized wave that overwhelds the service. Randomizing the delay helps diffuse the traffic and prevents a self-sustaining cycle.\n\nAnother important measure is the use of circuit breakers. After consecutive failures, the service should stop calling the problematic dependency altogether and fail fast. This approach, though counterintuitive, allows the downstream service to drain its queue, facilitating a faster recovery. Dropping requests immediately when their deadlines cannot be met is another effective strategy. Admission control prevents degraded services from becoming dead ones by shedding load at the entry point.\n\nThese strategies are particularly relevant in the context of agentic systems, where retries involve full model round trips that consume significant resources such as GPU seconds and cache memory. The cost of retries in such systems has multiplied, making the amplification factor even more pronounced. For instance, a three-attempt policy combined with independent retries by models can result in exponential traffic amplification, leading to self-sustaining failures.\n\nIn conclusion, the principle of avoiding trouble by not troubling it holds true in the design and operation of distributed systems. By adopting restraint in retry policies, employing backoff strategies, utilizing circuit breakers, and implementing admission control, engineers can prevent self-sustaining failures and ensure that systems recover promptly. These practices prevent the exponential amplification of load during faults, allowing services to function effectively even under degraded conditions.",
  "summary": "Learn how retry storms amplify failures in distributed systems and AI agents, and how retry budgets, backoff, jitter, and circuit breakers help.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}