What loss.backward() Actually Does: A Deep Dive into PyTorch Mechanics

What loss.backward() actually does

I built microcrad, a simple automatic differentiation engine inspired by Andrej Karpathy's micrograd, to reveal exactly what happens when you call loss.backward(). By breaking down local derivatives and the chain rule, I show how reverse-mode automatic differentiation computes gradients for all parameters in a single sweep, making neural network training possible.

"This asymmetry is the entire reason neural networks are trainable at all."