Patching Zsh to Crash: How I Found the History Truncation Bug
Tracking down a Zsh history data loss bug
For years, Michael Stapelberg occasionally found his Zsh history file truncated to only very old entries. After ruling out inotify and fatrace, he used bpftrace to trace Zsh's file operations, revealing that Zsh was rewriting the history file with fewer lines than expected. To pinpoint the bug, he patched Zsh to crash when writing a truncated history file, then analyzed the core dump. The investigation led to a fix in Zsh 5.9.2. The article details the debugging process and includes appendices on an exported HISTFILE footgun and whether AI can find the bug.
Spoiler: ultimately, patching Zsh to make it crash loudly and analyzing the crash’s core dump was the winning strategy!
- mmh0000
I have nearly a decade of zsh history. Reading that article I came to the conclusion that I may have been hit by that bug in the past but I haven’t noticed.
Then I kept reading and the author mentions accidentally exporting HISTFILE[1] and I screamed in terror and ran to my computer as I realized a mistake I’ve been making for…ever.
I am now both happy and sad I read this article.
[1] https://github.com/stapelberg/configfiles/commit/32dcda0f49a...
- pratyahava
this is an example of overcomplicated solutions that could be avoided with the tool we already have for years - file system - why do this "heroic" effort of "cleverly" putting everything in one file from many sessions if we could just write command history of every session into a new separate file in a directory and read all history files from the dir instead of reading one file.
- chillpenguin
I ran into this a few years ago (or I ran into a very similar bug). I started backing up my history file regularly because it was so annoying to lose all that history. I'll keep backing it up, but this fix is appreciated!
- conferza
A very interesting investigation, thanks for sharing
- myshapeprotocol
Tracking down subtle data loss edge cases in shell history internals requires absolute precision. Great debugging write-up.