Our deploy script restarted twelve servers in three minutes and systemctl said yes to every one
Our rates API runs on twelve virtual machines as a systemd service. The deploy script is older than most of our team. For each host in turn it copies the new jar, runs systemctl restart over ssh and moves on to the next host if the command succeeded. A failure stops the loop, which is meant to leave eleven healthy servers when the first one goes wrong. On a Tuesday in October a release added a…
Our rates API operates on twelve virtual machines using systemd services. The outdated deploy script copies the new jar file to each host, attempts a restart via ssh, and proceeds to the next host upon success. A failure stops the loop, intended to leave the other servers functional. A recent release introduced a new configuration setting for a currency feed, added to the staging environment but not production.
During deployment, the script took three minutes to complete, successfully restarting all twelve servers. However, systemd responded with success immediately after forking the process, even before the application could read its configuration. This meant the missing setting caused the application to fail during startup, leading to the script moving on to the next host.
The loop's safety check, checking the service's readiness endpoint, proved too early, as the application was not yet ready. To address the issue, the script now waits for the readiness endpoint to return a successful response within ninety seconds or halts the entire deployment. The team also plans to add a configuration check against the application's list of required settings before deploying any jar files in production.
While systemd confirms that the restart was requested, the application's actual readiness should be verified through its own interface.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
