How a Flaky Test Exposed a Redis Client Use-After-Free Bug
We tracked down a mysterious memory corruption bug in our Redis client after a flaky test suite blocked our deployments. What started as a race condition investigation led to a 'double free' error and eventually a segmentation fault. By leveraging our Test Engine reliability scores and analyzing a core dump with ASAN, we discovered a use-after-free vulnerability in the hiredis C extension caused by a threading issue.
Memory corruption bugs such as these can often take months to fix, because having a core dump generated by a segfault only tells you that memory was corrupted at some point in the past, but gives you few clues as to how it became corrupt.
- toast0
> Harder still is when the cause of the bug happens prior to its manifestation.
I've yet to debug a big where the manifestation happens prior to the cause... Sounds even harder. :p
- StilesCrisis
Paragraphs and paragraphs of shaggy-dog setup, then "a coworker enabled ASAN and reported a bug [here]" with no elucidation. Disappointing.
- serious_angel
Thank you very much! This is great read! Though, no actual steps mentioned, including the:
> From this, he ascertained that the exception was triggered by a call to the C function memmove. While the source and destination addresses for this call looked sane, the size field was set to...
I do get the most of these stated, but it would be awesome to know more about how he did actually find it, and with what toolset. Was is gdb, Valgrind, JetBrains, IDA Pro, or some another general/complex dump viewer/toolset?
And, the interactive miracle, at the top of the website is freaking marvelous... It reminded me of the articles of the fluid simulation algorithms of Yusuke Endoh! Heartfelt gratitude to the developer, the artist who implemented it!
Apparently, behind the scene, the fluid colors are contained in an array where every single color has a defined meaning!
// categories in STACK order: index 0 at the floor → 4 at the surface
const CATS = [
{ key: "Code", rgb: [133, 232, 157] },
{ key: "Craft", rgb: [179, 146, 240] },
{ key: "Lessons", rgb: [249, 117, 131] },
{ key: "Story", rgb: [255, 171, 112] },
{ key: "Opinion", rgb: [158, 203, 255] },
];