Our label service ran out of local ports at four hundred and seventy requests a second

On a Monday in October, the label service of a retailer began to falter, handling roughly one request out of every five at its peak. Despite CPU usage remaining under thirty percent on its virtual machines, the Python application faced an error: [Errno 99] Cannot assign requested address. The service, which requested label PDFs from a rendering service through an internal load balancer, created a new TCP connection for each label, closing it immediately after the response.

The TIME_WAIT state of Linux kept these connections occupied for sixty seconds, limiting the number of new connections a machine could handle to about four hundred and seventy per second to one destination. With Linux allocating local ports from 32768 to 60999, a single machine could only open about four hundred and seventy new connections a second before reaching its limit.

This Monday, the service exceeded that limit by more than five hundred requests, causing the process to fail immediately once the range was depleted. The issue stemmed from the helper function using a new TCP connection for each label, exhausting the available ports and resulting in failed connections. The fix involved sharing a single requests Session across the process, with a connection pool sized for the number of workers, reducing TIME_WAIT sockets from around twenty seven thousand to under a hundred.

This adjustment allowed the service to operate more efficiently, with node_exporter's sockstat collector now alerting when any machine surpasses ten thousand TIME_WAIT sockets.