We migrated the receiver but forgot the return trip
We migrated the receiver but forgot the return trip Same incident, different lesson. (If you're counting, this is the day our infrastructure decided to teach a whole curriculum — see also your webhook is fine, your DNS cache isn't .) We'd moved the webhook receiver to a new server. Inbound delivery verified: webhooks landing, 200s going back, logs clean. Migration complete, high fives all around.…
The incident involved migrating a webhook receiver to a new server, but neglecting to establish the return connection. While the receiver was functioning correctly, the processor on a separate machine never received any of the incoming events. This occurred due to the absence of a sync job that would transfer the files from the new server to the processing machine.
The problem stemmed from a lack of comprehensive end-to-end verification; only checking the inbound and outbound endpoints failed to reveal the issue. To rectify the situation, the team had to recreate the sync job within five minutes. The key takeaway is the importance of verifying the complete chain of components, not just their endpoints, during a migration process.
Both ends appearing healthy does not guarantee a fully operational system; it's crucial to observe the entire flow, from the source to the destination, to ensure everything is working as intended.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
