Least-Squares to the Rescue for Exploding Variance in Log-Sum-Exp Estimation

Exploding variance of means of exponentials: least-squares to the rescue

Estimating log-sum-exp functions is plagued by variance that explodes exponentially with the scale of the potential. Francis Bach shows that a weighted chi-square variational formulation turns the problem into a continuum of least-squares problems solved in parallel, yielding closed-form spectral estimators for relative density and KL divergence. The key is an integral representation that connects f-divergences to operator convex functions, sidestepping the instability of empirical averages.