When a node loses its lease anyway: a takeover, second by second
I am building a cloud trade copier for MetaTrader. Each follower account has one broker session on one node, and that node copies the master's trades onto it. In an earlier post I described how a node leases its accounts from Postgres: a node_id and a lease_until on the account row, FOR UPDATE SKIP LOCKED to claim, and a node that stops sending orders 15 seconds before its lease ends. That post…
When a node loses its lease, the takeover process unfolds second by second. Initially, the node continues to trade but cannot prove its ownership. At t=20, a failed renewal cycle occurs, and the node logs an error, continuing to send orders until t=45, when its lease status changes to false. Subsequently, work is blocked, preventing order duplication.
At t=60, the lease expires, and another node can claim the accounts. In the 60-80 minute window, the claiming node takes ownership, while the original node logs the loss of its accounts. Upon reconnecting to the database, the original node may find that another node already owns its accounts, leading to a disconnect. During the lost period, no trades occur, representing the cost of the takeover.
The master continues trading unaffected, while the failed node drops its work. When the master resumes, it drops events that occurred during the gap, ensuring no incomplete history is trusted. The reconciliation process compares the master and follower states, generating actions based on the current positions and pending orders.
Late opens are prevented by checking for tagged orders, ensuring the master's trades are accurately copied.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
