Gradient Descent Can Train Any Neural Network—If You Redesign It
Universality of Gradient Descent Neural Network Training
A new paper proves a surprising universality result: for any neural network that can be trained by some algorithm, there exists an extended version that can be trained by plain gradient descent. The construction is theoretical, not practical, but it sheds light on the power of meta-learning and network design.
If, for a given network, there is any algorithm that can find good network weights for a classification task, then there exists an extension of this network that reproduces these weights and the corresponding forward output by mere gradient descent training.