Java Benchmarks Never Hit Peak Performance, Study Finds

Experimental Evaluation Methodology for the Era of No Steady Performance [pdf]

Researchers from Charles University, Università della Svizzera italiana, and Oracle Labs analyzed long runs of the Renaissance benchmark suite on the JVM and found that even large workloads fail to reach a steady state of peak performance. They argue this is an inherent property, not a fault, and propose a new methodology that models performance as a random process with steady segments separated by sudden changes. Their confidence intervals miss the true performance up to 24% less often than existing methods.