Why we stopped retrying on timeout and built reconciliation instead
Early in StareBrain’s action dispatch layer, we had a simple rule: if an action times out, retry it. It felt safe. It felt standard. It was wrong. The problem isn’t the retry. The problem is what a timeout actually tells you — which is almost nothing. What a timeout means (and doesn’t mean) When a dispatch call times out, you know one thing: you didn’t get a response in time. That’s it. You don’t…
StareBrain, an action dispatch layer, previously had a simple rule: if an action timed out, retry it. However, this approach proved to be wrong. A timeout only indicates that a response wasn't received in time; it doesn't provide any other information about the action's status. Retrying in this state doubles the number of dispatches in flight without resolving the ambiguity. This could lead to multiple bookings, charges, or messages being created for a single user request.
To address this issue, the team stopped treating timeout as a signal to retry and began treating it as a signal to reconcile. They introduced a state model with two axes: `DeliveryState` and `AcceptanceState`. Delivery tracks whether the action reached the provider and received a response, while Acceptance tracks whether the provider processed the action correctly. These are independent, with Delivery confirming the provider's response and Acceptance verifying the provider's correct processing.
When an action times out, its Delivery state remains PENDING, indicating that the action hasn't been confirmed yet. The team assigned a new status, `DENIED_UNRESOLVED`, to actions that cannot be confirmed programmatically. This status never changes to success due to a timeout; it remains unresolved until a human review or a successful reconciliation pass clears it.
The team recognized that some providers have replication lag, meaning that a record written during a slow dispatch might not be queryable for several seconds after the call returns. To handle this, they introduced a concept called `eligibleForReconciliationAt`, which is set at dispatch time based on a consistency window duration. Reconciliation jobs only run after this gate, ensuring that the provider's query surface is used correctly.
Providers are categorized into different capability tiers based on their query capabilities. FULL supports idempotency keys and query endpoints, allowing for full reconciliation. RECONCILABLE providers have a query endpoint but no idempotency key support. PARTIAL providers only return artifact IDs, while BLIND providers offer no reconciliation possibility, requiring human review instead.
The switch from retry-on-timeout to reconciliation-on-timeout resulted in zero duplicate actions from network blips, a clear paper trail for unresolved actions, and provider integrations graded by their actual capabilities. Although this approach requires more infrastructure, state management, and explicit handling of ambiguity, it ultimately provides a more reliable and correct solution compared to pretending timeouts mean failures and retrying.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.