Why the Mean Means Nothing: Visualizing Latency Beyond Single Numbers
The mean means nothing: data visualization to debug a latency problem
I discovered that relying on mean latency masked a complex reality where typical requests sped up while tail latencies doubled. By using Cumulative Distribution Functions and shift plots instead of simple averages, I revealed how a caching rollout created two distinct performance populations. This experience taught me that a single statistic often fails to tell the whole story, urging engineers to visualize the full shape of their data distributions.
Two CDFs that cross are the unmistakable signature of a change that no single percentile can summarize, because the sign of the effect depends on which percentile you ask.
- jldugger
It's little surprising that the author was doing perf work and not already comparing distributions. As the rest of the article outlines, you learn a lot more with more data!
It's complicated, but really worth learning how prometheus and grafana heatmaps combine if you want dashboards for real time service data. Multimodal distributions are basically the expected outcome given all the caching done in distributed systems.
- jeffwass
An interesting statistical example if you haven’t seen it before is the Anscombe Quartet :
https://en.wikipedia.org/wiki/Anscombe%27s_quartet
Four sets of X,Y datapoints that have exactly (or very close) common statistical parameters (mean, variance, correlation, linear regression, R^2), but with vastly different spatial distributions and “behavior” when looked at visually.
- danbruc
Instead of the CDF I like to use one minus the CDF, the fraction of requests not yet completed at any given time. Then you can make it a log log plot showing the entire tail with details invisible in the CDF because in the CDF the tail is essentially a horizontal line at one.