The silent retry bug killing ASP.NET Core reliability (and how idempotency keys save you)

When working with distributed systems in C# and ASP.NET Core, network issues like timeouts and service unavailability are common. The conventional solution is to use Polly, a library for resilience and transient fault handling, by adding retry policies. However, retries without idempotency can lead to data inconsistency. For instance, retrying a POST or PUT request might change the system state, leading to issues like double charges in a payment system or duplicate orders in an e-commerce platform.

This problem is referred to as the "duplicate-state problem," which is a hidden threat to system reliability.

Most retry strategies, including those from Polly, Hangfire, and Azure Service Bus, operate on the "at least once" delivery guarantee. This means they will retry if something seems to have failed, but without idempotency, a successful first call followed by a failed second call (or vice versa) can result in inconsistent data.

The solution to this issue is the use of Idempotency Keys. This involves generating a unique identifier (typically a GUID) for the operation, attaching it to the HTTP request (often in the Idempotency-Key header), and having the server check if this key has been seen before. If it's new, the server executes the business logic, saves the result, and records the key-result pair with a Time-To-Live (TTL). If the key is already present, the server skips the business logic and returns the stored result.

In ASP.NET Core, implementing this involves checking for existing results in a fast, distributed cache like Redis, executing the business logic if no existing result is found, and storing the key-result pair in the cache. It's recommended to encapsulate this logic in middleware or a filter for uniform application across all endpoints.

Using Polly, you can ensure that every retryable HTTP call includes the idempotency key, either by adding the header automatically for specific clients or by ensuring clients always provide it. For background jobs in Hangfire or Azure Functions, the same principle applies—ensure the underlying HTTP calls are idempotent to maintain data integrity during retries.

In summary, resilience in distributed systems isn't just about avoiding crashes; it's about maintaining data integrity during failures. Implementing Idempotency Keys for all state-changing endpoints and using a distributed cache to store these keys can safeguard your system against the "duplicate-state problem." Always test your retry logic by simulating scenarios like timeouts after successful writes to ensure your system doesn't create duplicate data.