Inode Exhaustion: The Disk Full Error That Makes No Sense

The inode exhaustion issue occurs when a disk appears full due to a lack of available inodes, rather than a shortage of disk space. An inode is a data structure that stores metadata about each file, such as permissions, owner, timestamps, and pointers to the data blocks on disk. Every file, directory, and symlink requires an inode to function correctly. The problem arises when a large number of small files accumulate, consuming all available inodes.

The classic scenario involves a production server hosting a Node.js application. The disk showed 85% usage, but the inode count was at 100%. This contradiction confuses engineers, who must investigate further. Tiny files, such as those created by npm installs, webpack cache entries, or log rotation, contribute significantly to inode exhaustion. Misconfigured logrotate settings, ephemeral session files, and PHP sessions can also exacerbate the issue.

To diagnose inode exhaustion, first confirm the problem by running df -i. If any partition shows 100% inode usage, the issue is identified. Next, find the largest directories using the du command and prioritize directories with many files, not necessarily the largest by size. Once the suspect directory is found, check if the files are small (under 1KB) using the find command. Deleting these files frees up inodes and resolves the issue.

Prevention involves proper logrotate configuration, which compresses and rotates old logs instead of accumulating them. A cron job can warn administrators when inodes reach critical levels, reducing panic and downtime. In cases where the inode count is too low for the workload, options include resizing the filesystem, moving data to a larger partition, or recreating the filesystem with higher inode density.

In summary, inode exhaustion is a common issue caused by a lack of available inodes. By understanding the causes and implementing preventative measures, sysadmins can avoid this frustrating "No space left on device" error.