My server didn't go down. Chasing the false alarm hardened everything else.
On November 7, Discord pinged me: ๐ด Container(s) Down: backend_api. I opened my site on my phone and it didn't load. That looked like proof. It wasn't. I SSHed into the droplet and read the logs. Every container had been up the whole time. My phone's 4G had dropped out, and the alert was false. That should have been a five-minute investigation, but it turned into two days and 13 pull requests,โฆ
On November 7, Discord informed the reporter that one of the containers was down: backend_api. The reporter opened the site on their phone, and it didn't load. However, upon SSHing into the server and checking the logs, they discovered that every container had been running the whole time. The issue turned out to be a false alarm caused by the reporter's 4G connection dropping out.
What initially seemed like a five-minute investigation expanded into two days and 13 pull requests, as each fix exposed the next issue that had gone unnoticed. This post chronicles the problem, how each fix was verified on the live server, and the surprising insights gained along the way.
Written by urgent.news from Dev.to's reporting โ not their text. Machine-written โ may contain errors; check the original before relying on it.