Retry context: building observability into retry decisions

Making retry decisions trustworthy requires visibility into what's happening. A retry attempt appears identical whether it's recovering from a transient network glitch or getting stuck in a loop that burns your budget on a permanent failure. Without tracking the details of each attempt, your retry logic is essentially guessing.

Idempotency keys are the foundation. These unique tokens attached to requests allow the system to recognize if it has seen a request before and return the cached result instead of repeating the work. Idempotency keys don't directly relate to retries, but they are crucial for making retries safe. An idempotency key is a stable identity that verifies if an operation is safe to retry.

For example, a stable idempotency key should remain the same for a retry window of at least one second while being different for each retry attempt.

To correlate retry attempts, you need to track which attempt belongs to which logical operation. This is where retry-attempt correlation comes into play. By logging each attempt with details like latency, status, and error (if any), you create a chain of custody that allows humans or monitoring systems to follow the story of a single logical operation through its retries.

This correlation helps answer questions like whether all attempts failed for the same reason or if they failed differently. If multiple attempts encounter the same error, it suggests a permanent limit rather than a transient issue. On the other hand, if early attempts fail with a timeout and subsequent attempts succeed, it indicates the earlier transient failures were indeed transient, validating the need for retries.