Broken Retries: Why 'Just Try Again' Kept 5 Automation Lanes Dead for 63 Minutes

Five automation lanes went down on the same morning, all of them failing and retrying three times each. However, the same failure occurred each time, leading to a total of 63 minutes of downtime. The source material explains that the design of retries, which assumes the next attempt will succeed, does not work in all cases of failure.

The article points out that when the number of running jobs increases, failures that break this assumption will inevitably appear. In the SNS automation environment, where the author operates, five automation lanes failed over two days, with different causes for each failure. The source material highlights that retry fixes came in from these lanes at once, but the results were varied.

The author describes three types of failures: deterministic failures, such as a 63-character title issue, which will always produce the same result; moderation blocks caused by content; and slot shortages, which can only be solved by waiting for a slot to free up.

The article emphasizes that a retry implementation without classification of failure types counts fixable and unfixable failures identically, leading to a failure mode where similar incidents are logged under different names, causing confusion in monitoring and troubleshooting. The author argues that a successful automation environment requires self-correction and the ability to keep running while the developer is asleep.

The author concludes by presenting three questions to guide retry design: (1) Will waiting fix it? (2) When do you cut it off? (3) What do you change before the next attempt? The author emphasizes that the original "just try again" implementation fails to address these questions, leading to inefficient incident recovery.