Exactly-once deposits: a broker cannot deduplicate a reorg
A service that credits user balances from on-chain deposits has to tell its consumer about each deposit exactly once. The usual reflex is to push the problem onto the transport: turn on deduplication in the queue, set a dedup id, move on. That solves the wrong half. A broker deduplicates the delivery of a message you already decided to send. Nothing in it deduplicates the decision , and on a…
Exactly-once deposits refer to the requirement that a service crediting user balances from on-chain deposits must communicate each deposit to its consumer exactly once. The typical approach is to utilize deduplication in the queue by enabling it and setting a deduplication ID, but this only addresses half of the problem. The other half lies in the decision-making process, which remains unaffected by the broker's deduplication.
On a blockchain, the decision remains constant, making deduplication on the producer side necessary.
When building custodial wallets, on-chain payment systems, and fiat-to-crypto onramps, the author encountered numerous exactly-once bugs over three market cycles, all of which originated from the producer side. The issue arises from the way a deposit watcher polls for new deposits, re-reading a range of blocks that overlap the previous one to detect any missed blocks. This results in the same transfer being presented to the code multiple times, regardless of any restarts, resyncs, or reorganizations.
Brokers can aid in deduplication only if they receive the same deduplication ID each time. However, determining whether a transfer is the same as a previously reported one is a state problem. If a stable ID can be computed, the broker is not required; if it cannot, the broker cannot help in deduplication. The payment key should represent the payment itself, independent of when it was learned about and where it was mined.
Chain-watch library maintains this key as DepositKey, which is derived from the (tx_id, output_index) pair of the transaction output where the money landed.
Most implementations fail to model the unique identity correctly, leading to issues with broker deduplication. The identity should name the payment itself, instead of depending on when it was learned about or where it was mined. A transaction ID and output index pair (tx_id, output_index) can provide a stable key that holds across polls, duplicated blocks, restarts, and reorganizations, making it the natural unique constraint downstream.
Deduplication windows are measured in time, while finality is measured in block depth. Even if a broker deduplicates using a key supplied by the consumer, it does so within a specific time window, typically minutes. Chain finality, however, is expressed in block depth, which differs between chains and assets. A resynchronization can potentially replay an entire year's worth of history within the time it takes to read it, making it impossible to configure a time window that guarantees never to report a key again.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
