Stop Running Database Migrations in initContainers
How we moved Flyway into a Helm hook that runs once, before the deploy, and works the same in Argo CD Our app ran 6 replicas. On every deploy, 6 pods tried to migrate the same MySQL database. We had put Flyway in an initContainer. It looked clean: the migration runs before the app starts, so the app always sees the right schema. It's a common pattern, and for a single pod, it's fine. With 6…
Running database migrations in initContainers has proven problematic when deploying applications like Spring services on MySQL using Helm, Argo CD, and Flyway. The issue arises because every replica runs the migration every time a new pod is created, restarted, rescheduled, or when the Horizontal Pod Autoscaler (HPA) scales up. This leads to resource contention, long waiting times for new pods, and difficulties in identifying migration failures.
To resolve these issues, the team moved Flyway into a Helm hook that runs once before the deploy, working the same way in Argo CD. This change eliminates the need for initContainers and simplifies the migration process. The solution consists of three parts: a migration image containing the SQL files, a Job with Helm hook annotations, and the application with Flyway turned off.
The migration image is built using an official Flyway image and includes the SQL files in the image. The Job runs as a Helm hook before any other changes in the release, ensuring that migrations are applied once per release. The Job includes necessary environment variables, such as database connection details, and attempts to connect to the database with retries in case of connection issues.
The application's Flyway is disabled by setting the `SPRING_FLYWAY_ENABLED` environment variable to `false`. This ensures that the application connects to the database without attempting to run migrations at startup.
When a deploy occurs, the order is as follows: Helm creates the migration Job, Flyway applies any new migrations, and then Helm updates the Deployment. New pods from the HPA do not touch Flyway at all, as it has already been completed. If a migration fails, the Helm upgrade stops, allowing the existing pods to continue running, and the logs from the failed Job can be reviewed for troubleshooting.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
