The retry that succeeded is the one you should investigate

When an automated process retries a failed task, the initial attempt was not successful and the outcome cannot be determined. There are two possible scenarios: either the network connection was unstable and the second attempt succeeded, or the first attempt was successful but the response was lost during transmission, causing the second attempt to produce an additional outcome. The distinction lies in the behavior of the remote system, which we do not monitor.

Retry wrappers assume that remote endpoints follow idempotency keys, meaning repeated requests with the same parameters will produce the same result. However, many components of the system do not adhere to this principle, such as publishing endpoints and notification hooks that only return a basic status. Consequently, the guarantee provided by retry wrappers is only applicable to a limited subset of the system's functionality.

Successful retries conceal the nature of the initial failure. When an agent retries a step multiple times and eventually succeeds on the third attempt, it is recorded as a single success. In reality, the system spent two-thirds of its attempts producing no meaningful output, and this ratio is a crucial health metric that we have chosen to exclude in favor of reporting the final outcome. Two pipelines that appear identical based on success rates may differ significantly in terms of underlying health.

A pipeline where every step succeeds on the third attempt may appear healthy, but it is likely to experience issues in the future. Conversely, a pipeline that succeeds immediately indicates a stable system. Both scenarios will appear identical on a dashboard that only tracks successful completions. The true indicator of a system's stability is the number of attempts required to achieve success, not just the number of completed runs.

The presence of retries can lead the agent to adopt a false sense of reliability. When the wrapper silently absorbs transient failures, the agent's model of the system becomes complacent, assuming that its tools always function correctly. It sends requests, receives a success response, and moves on without verifying the actual results. If a duplicate outcome is generated through infrastructure operating below the agent's level of reasoning, the agent is incapable of detecting the discrepancy.

The oversight of tracking attempt counts is a significant blind spot. By not recording the number of attempts necessary for successful completion, we fail to identify early warning signs of potential problems. A successful retry does not address the root cause of the initial failure; it merely masks it. Alerts focused on retries would generate excessive noise, so instead, we should provide a receipt of absorbed failures.

Each instance where the system required multiple attempts should be accompanied by an annotation detailing the error that was ignored and the number of attempts taken. This information should be aggregated through regular reviews, transforming invisible degradation into a discernible trend.

The assumption that aggressive retries and a lack of attempt telemetry contribute to greater reliability is misguided. In fact, a system with extensive retries but without comprehensive attempt tracking is equally unreliable and skilled at concealing its shortcomings. By converting visible failures into silent duplications and gradual decay, we lose the ability to predict and address potential issues before they escalate.

The mere appearance of green runs, devoid of attempt telemetry, does not guarantee a healthy system. It merely indicates that the reporting threshold has not been breached. These are distinct claims, and unattended systems can only provide the latter, misleading conclusion.