Cross-Validation From Scratch: A Surprise at n=100

Cross-Validation From Scratch and a Surprise at n=100

Textbooks claim LOOCV has the lowest bias but highest variance compared to 10-fold and 5-fold CV. I coded K-Fold CV from scratch to test this on simulated data. At n=1000, the claim holds, but at n=100, the order flips—LOOCV shows negative bias and higher variance, contrary to expectations. The post also explores recovering the true data-generating process via CV and notes that the slope from regressing y on predictions is less than 1 due to attenuation bias.