Finding zombies in our systems: A real-world story of CPU bottlenecks
Pinterest's ML training jobs on Ray were crashing due to intermittent network connectivity issues, traced to ENA network driver resets caused by CPU starvation. After months of investigation, the team discovered that a single CPU core could be saturated for seconds, leading to resets. They used profiling tools like perf and mpstat, and a creative experimental setup with gProfiler-like time-travel profiling, to finally identify the root cause: a 'zombie' process consuming CPU. The story highlights the challenges of diagnosing performance bottlenecks in distributed systems.
We realized that for CPU starvation to happen, it may take as little as one CPU core to be heavily utilized and block an unlucky network thread that was scheduled onto that core.
- troelsSteegin
This was a nostalgic pleasure to read in contrast to all the AI implications and tools posts. That said about AI, this as a case study would make an interesting benchmark.
Especially noted that the root cause here was pegged because somebody else had blogged about it. Our "digital commons" should be valued as other than an externality. Edit: by that I think that "the internet should be free" but that commercialization through derivative work should compensate authors. That's off-topic, so, nevermind.
- alain94040
Interesting story.
I don't know anything about the Linux scheduler, but if one core out of 96 is busy chasing after those ghost allocations, why would a network thread not use one of the other 95 cores, and instead be stuck for seconds? Unless that one core is blocking the entire kernel, but then that would be a lot more visible instantly.
- 4fterd4rk
I know this is a hopelessly naive question but what is Pinterest even doing with all this AI training?