Tracking down a Zsh history data loss bug

In the article "Tracking down a Zsh history data loss bug," the author explains how they investigated a problem where commands they believed they had executed were no longer present in their Z shell history file (.zsh_history). This issue persisted despite attempts to restore the history from daily backups. The author noticed that the .zsh_history file contained only old entries, with newer entries missing.

To diagnose the problem, the author first considered using file system change monitoring tools like inotify or fsevents to identify any program that might be truncating or modifying the .zsh_history file. However, they found that these tools did not provide the necessary process IDs (PIDs) associated with the events. They then turned to fatrace, which displayed the process name and PID.

The author discovered that Zsh was rewriting the .zsh_history file incorrectly, reading fewer lines than usual and then writing them to a new file, .zsh_history.new.

The author then decided to study the code of Zsh to understand why the readhistfile function was not reading the full history file or the savehistfile function was not writing the full history file. They found that the savehistfile function was writing out a shorter history file because the readhistfile function left it with a shorter history. This led to the discovery of the bug, which was subsequently fixed in Zsh 5.9.2.