A migration merged on Tuesday was never going to run in production
Two developers on the billing team each wrote a database migration in the same week. The first one, adding a column for invoice language, became V41. The second one, an index, became V42. The index was merged and deployed on Monday. The invoice language change was merged on Tuesday and deployed that afternoon. Sixteen days later, the first customer to pick a language for their invoices got an…
On Tuesday, two developers working on the billing team each submitted a database migration in the same week. The first migration, which added a column for invoice language, was version V41. The second migration, an index, was version V42. On Monday, the index migration was merged and deployed. However, the column for invoice language was not merged until Tuesday and was deployed that same afternoon.
Unfortunately, the issue only came to light sixteen days later when the first customer selecting a language for their invoices encountered an error page. The reason for this error was that the column for invoice language did not exist in the production environment. Flyway, the migration tool used, records each migration it applies and typically refuses to start if it detects a migration with a lower version than the highest one already applied. This is a safety measure to prevent historical data from being overwritten.
However, a year ago, during a hotfix, someone had set Flyway to ignore such migrations. This setting was intended to be temporary, but it remained active. As a result, when production saw V41 after V42, it decided that the V41 migration was simply in the past and skipped it without any warning. The application still started functioning correctly, and no other code was affected by the missing column.
To resolve the issue, the V41 migration was applied manually that morning. After realizing the mistake, the ignore setting was disabled, so any migration arriving out of order would now stop the deploy immediately, alerting developers in the first environment it encountered. Migration versions are now based on timestamps taken when the file is created, and a check in the merge queue will fail any pull request whose new migration is older than the newest one already on the main branch.
Additionally, after every production deploy, a job compares the list of applied migrations in production with the list in the code to alert on any discrepancies. This job previously found a migration from 2023 that had been skipped, leading to the current system in place. The refusal of the system to start due to an out-of-order migration was initially difficult to resolve during a hotfix, as it was the only part of the system that understood the proper order of migrations. The refusal was a consequence of the temporary setting being left on and forgotten.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
