How an Eventually Consistent DynamoDB GSI Let Mobile Retries Create 1,400 Duplicate Orders
14:07 UTC. PagerDuty fires on fulfillment-queue-depth , 340% of baseline and climbing. Checkout's on-call pulls up the dashboard expecting a stuck consumer or a slow downstream call. Instead she finds the queue isn't slow, it's just full of orders nobody placed twice on purpose. the scramble First theory: the mobile app's retry logic, rewritten three weeks ago to handle flaky connections more…
On February 14th, PagerDuty raised an alert about the fulfillment queue depth being 340% of its baseline. The fulfillment engineer expected a stuck consumer or a slow downstream call, but discovered a huge backlog of duplicate orders. The initial theory was that the mobile app's retry logic, which had been rewritten three weeks ago, was double-submitting requests.
However, analyzing the client-side request logs revealed that each order ID had only one tap, one request, and one Idempotency-Key header that was reused correctly across any retries.
The second theory, which took fifteen minutes to rule out, was that API Gateway was retrying a Lambda invocation that looked like it timed out, but actually succeeded. However, CloudWatch showed that every invocation completed in 180-240ms, well within the 29-second timeout, and API Gateway's own access logs confirmed that there were no retries at that layer.
The real culprit was discovered when someone noticed that every pair of duplicate orders shared the same Idempotency-Key attribute but had different order_id values. This inconsistency led the team to investigate the idempotency check itself. The checkout Lambda's create-order path was believed to handle idempotency correctly by looking up the key and returning an error if it existed, otherwise writing a new order.
However, the team realized that the Query against the Global Secondary Index (GSI) that the Lambda used for the idempotency check was not carrying any consistency guarantee. GSIs only support eventually consistent reads, and there was no way to request strong consistency for a GSI. This lack of consistency was not an issue until the mobile carrier experienced a blip in one metro area, causing the GSI propagation delay to spike to 400-600ms during the same window.
To reproduce the issue in staging, the team fired five identical create-order calls 250ms apart, using the same Idempotency-Key. The staging environment confirmed that two legitimate-looking requests, 300ms apart, could both query the GSI that hadn't caught up from the first request yet, resulting in duplicate orders.
The fix for this issue was not to improve the GSI's consistency, as there is no option for strong consistency on a GSI. Instead, the solution was to move the correctness check off the index and onto the base table, where DynamoDB provides atomicity. The updated code now uses a conditional write against the base table, which is atomic and guarantees that only one invocation succeeds, while the other receives a ConditionalCheckFailedException.
This approach ensures that even if there is a replication lag, the system will still maintain data integrity and prevent duplicate orders.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
