Why the Mean Means Nothing: Visualizing Latency Beyond Single Numbers
The mean means nothing: data visualization to debug a latency problem
I discovered that relying on mean latency masked a complex reality where typical requests sped up while tail latencies doubled. By using Cumulative Distribution Functions and shift plots instead of simple averages, I revealed how a caching rollout created two distinct performance populations. This experience taught me that a single statistic often fails to tell the whole story, urging engineers to visualize the full shape of their data distributions.
Two CDFs that cross are the unmistakable signature of a change that no single percentile can summarize, because the sign of the effect depends on which percentile you ask.