Address Space Exhaustion in Production
I went looking for why none of our registered users were using the product. I found something worse: for most of them, the product had simply stopped working. Address space exhaustion had been quietly breaking mailbox creation for weeks, and not one of our monitoring tools said so. One of them actively told us the opposite. This is the writeup of that outage. The bug itself is almost…
The email service stopped working for most registered users due to address space exhaustion. This issue had been quietly breaking mailbox creation for weeks without any monitoring tools detecting it. The bug was caused by a combinatorial space that filled up because nothing ever left it. The generator produced 36,000 possible combinations, and the creation endpoint attempted to generate a candidate address up to five times if a collision occurred.
Despite the number being clearly stated in the code, the service failed silently, with only 18 errors appearing in the analytics over six weeks. The API also returned hundreds of 503 errors, but these were not reflected in the funnel_error events or container logs. The failure rate went from undetectable to nearly 70% in a short period, highlighting the importance of monitoring the system comprehensively rather than relying solely on analytics.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
