HRT traces a network mystery with eBPF across its global compute cluster
Tracing Mystery Network Traffic with eBPF Across HRT's Global Compute Cluster
When a model training run slowed down, HRT engineers discovered that unknown traffic was saturating their WAN link. They built a system that combines service telemetry, eBPF kernel tracing, and network hardware byte counters to attribute traffic to specific users and programs. The culprit: a configuration error that caused thousands of jobs to read a file directly from a remote filer instead of using the local cache.
In retrospect, this was a simple mistake with an easy fix. Our tracing helped with the hard part: figuring out what the mistake even was.
- gandreani
This is a very nice use of eBPF and probably close to the imagined scenario the kernel developers proposed for this.
eBPF has a terrible name but it's absolutely wonderful tech. If you're running a database SaaS you can implement eBPF filters to do the accounting for billing based usage, regardless of which database you use!
To me that's irrefutable evidence that it's really well designed
- muststopmyths
Not familiar with NFS, but shouldn't it have been possible to attribute the mystery traffic by port number by sniffing ?
- cmdrk
The high volume site-to-site data transfers for running simulation etc reminds me strongly of the high energy physics computing world. no wonder so many HEP folks ended up at HRT!