I Kept Deleting Logs for 48 Hours. The Inodes Were Already Gone.
Have you ever watched a two-kilobyte write fail with No space left on device while df -h still showed free gigabytes? I did, and I spent the next forty-eight hours cleaning the wrong evidence. This is the reconstructed field notebook from that session, including the commands I ran, the ones that misled me, and the checklist I now run before I blame the disk. Nothing here is a benchmark, a quota…
In a puzzling incident, a Python worker was experiencing a recurring issue where it would fail to write a small JSON file due to "No space left on device." Despite df -h reporting ample free disk space, the worker continued to run into errors. The reporter, after a 48-hour investigation, discovered that the problem was actually due to inode exhaustion rather than a lack of disk blocks.
Initially, the reporter treated the issue as a log rotation problem, attempting to truncate and delete old JSON sidecars and worker logs. However, the issue persisted, sometimes failing on file number 20, and other times on file number 4. The reporter also tried removing large JSON files, restarting the process, and questioning whether the path was on a different mount than $HOME directory, but these actions did not resolve the problem.
The turning point came around hour 12, when the reporter realized that their cleanup script was actually making the situation worse. Their "safe delete" helper was copying each file to a trash directory before unlinking it, which, while cheap in terms of disk space, was expensive in terms of inode usage. This was because each copy created an additional inode, and with thousands of small files being processed, the inode count was quickly approaching the filesystem's limit.
Recognizing that the issue was not with blocks but with the number of inodes, the reporter finally ran the df -i command, which showed that the mount was indeed full of inodes. This revelation led the reporter to focus on file count, extension histogram, and cache directories as the areas to investigate further. It turned out that the noisy directories were not the logs that had been truncated, but rather caches and sidecars.
To avoid such a trap in the future, the reporter created a reproducible inode trap, creating many tiny files on a local directory to demonstrate how quickly the inode count could fill up. This demonstration, which should only be run on a throwaway directory on a filesystem the user owns, creates N (set to 20,000 by default) tiny JSON files and shows how df -i can move as blocks remain plentiful.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
