Idempotent Video Requests — Preventing Duplicate Generation When Retries Collide

The article discusses strategies to prevent duplicate video generation when retries collide in a fintech upload pipeline. It recommends using an idempotency key stored in the database as a durable record, treating video generation as a state transition rather than a fire-and-forget retry. This approach ensures one thumbnail job is attached to one audit trail while maintaining measurable moderation coverage.

The key record should contain information such as the client's account ID, the idempotency key, a content digest of the video, thumbnail dimensions, moderation policy identifier, response snapshot, and a content digest. It should not store raw card data but rather a pointer to the governed media store. The lifecycle of the record should be tracked as accepted, processing, moderated, ready, or rejected.

The article provides a TypeScript implementation of the core transaction, where the provider call happens after the row is committed, preventing two HTTP workers from claiming the same request. The outbox worker handles the provider job identifier and renewal of the lease, with a reconciliation query for operator decision if upstream API lacks correlation lookup.

Metrics for monitoring include duplicate-key conflicts, the age of the oldest processing row, and the percentage of generated thumbnails covered by the active moderation rule. A GET /media/requests/{receipt} endpoint and a clear processing state can help resolve support loops. The article concludes that three metrics are sufficient to start monitoring, emphasizing the importance of using pseudonymous hashes for keys instead of raw account secrets.