RipGrep musl Binaries Crash During Massive Concurrent Searches
RipGrep musl binaries occasionally segfault during very-large searches
I discovered that RipGrep binaries compiled with musl occasionally trigger a segmentation fault when scanning extremely large directory trees with high concurrency. The crash occurs inside the musl allocator during a calloc call made by opendir, specifically when searching for a string across millions of files. Reproducing this issue requires a massive dataset and running repeated search loops on a multi-core system.
The crashing line is an integrity assertion regarding heap metadata inside MUSL's mallocng, in a calloc call made from opendir.
- ndesaulniers
Heh, from the kernel patch: https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...
> I saw a fun bug report in ripgrep and a studious but pretty bad
AI-generated analysis
Referring to https://github.com/dfoxfranke/ripgrep-3494-analysis which I indeed thought "that's an awful lot written to have been written by a human."
Looks like that thread is from...today!
- Orphis
I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven't bothered replacing it with another more performant one.
mallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in multithreaded scenarios (and only just 8 threads). Switching to mimalloc improved performance by 20x, very close to what glibc offers by default, and just a bit under a glibc + mimalloc configuration.
I get that there's a real issue there and it's interesting (to some) to address it, but it should have never surfaced this way in the first place.
- dosman33
Anyone running ripgrep on a an HPC cluster against a large cluster filesystem needs to stop and redesign their workflow. This generates high amounts of small I/O which is the Achilles heel of any large cluster filesystem. You are exporting your workload onto the metadata mechanisms of the filesystem rather keeping it within the higher bandwidth capable memory subsystem on your cluster. It doesn't take but a couple users running these types of jobs simultaneously to bring a high-bandwidth filesystem to its knees. Just stop it already.
- hyperpape
The analysis of the kernel bug may be a better thing to link to: https://github.com/dfoxfranke/ripgrep-3494-analysis.
- sligor
So, why the bug triggers only with muslc and not other libc ?
- walk12111
I would normally suspect musl's thread stack size. Is the kernel bug confirmed?
- wild_pointer
wow, everything is broken lol
- villgax
No wonder search in codex is so a$$