Production Logged Zero Errors. Every Call Was Refused First.
On the morning of November 13, nine calls to our server failed in a row. Our logs for that morning had zero errors in them. No 5xx all day. Both of those things were true at once, and it took us most of that day to understand how. I build Since, which tracks the packages, AI models, APIs and runtimes your code depends on, tells you and your coding agent what changed and when it breaks, and opens…
On November 13, nine consecutive calls to the server failed without any logged errors. The monitoring system seemed fine, with no 5xx errors and all endpoints responding. However, the root cause was hidden in the status codes: every call was refused first, then accepted. The logs showed a pattern of 400 errors before successful 200 responses.
The monitoring system missed this pattern, as 400 errors were logged at the info level and not treated as errors. The issue stemmed from the connector using a newer protocol revision that the server didn't support. The server refused the call with a 400 error, and the client fell back to an older revision, which worked upon retry.
Additionally, public scans running on the server's main thread caused health checks to fail due to time-outs, leading to dropped connections. The incident highlighted the importance of proper logging and monitoring of status codes, as well as the need to align protocol revisions between the server and client.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.