Finding zombies in our systems: A real-world story of CPU bottlenecks

Pinterest's ML training jobs on Ray were crashing due to intermittent network connectivity issues, traced to ENA network driver resets caused by CPU starvation. After months of investigation, the team discovered that a single CPU core could be saturated for seconds, leading to resets. They used profiling tools like perf and mpstat, and a creative experimental setup with gProfiler-like time-travel profiling, to finally identify the root cause: a 'zombie' process consuming CPU. The story highlights the challenges of diagnosing performance bottlenecks in distributed systems.