The Leak That Won't Die: How We Broke Python's tempfile
Production crashes always happen suddenly. This time, the alarm was raised by the core C++ SDK developers. Their scheduled nightly build failed on the CI runner with a classic diagnosis: No space left on device . We ran out of disk space. A quick post-mortem revealed that the /tmp folder was bursting with thousands of temporary directories. Their names all started the same way: ude_xml_ . It was…
The production crash was triggered by a leak of temporary directories in the /tmp folder. The issue stemmed from the engine's handling of Doxygen, which created temporary folders using the tempfile.mkdtemp() function. If the parsing failed, the cleanup() function responsible for deleting the folders was never called, resulting in garbage folders accumulating in /tmp.
The leak was flaky and difficult to detect. In an attempt to solve the issue, the developers replaced mkdtemp() with tempfile.TemporaryDirectory, which was supposed to automatically delete the folder when the object was deleted from memory. However, the developers mistakenly relied on the Garbage Collector (GC) to handle the cleanup, a flawed approach.
The engineers rolled out the fix, but the problem resurfaced in the CI runners. The cleanup problem was due to circular references in the parsers that prevented the objects from being deleted immediately. The TemporaryDirectory objects accumulated in memory, causing a massive synchronous deletion of files from the disk, which blocked the main execution thread and caused timeouts in other services.
The solution was to use strict context managers for managing temporary folders, ensuring that the folder and cleanup() happened together. A janitor mechanism was also added to regularly clean up any orphaned garbage in the /tmp folder.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
