How You Could Have Invented Kimi Delta Attention from First Principles

You Could Have Come Up with Kimi Delta Attention

I walk through the evolution from standard softmax attention to linear variants like DeltaNet and Kimi Delta Attention. By treating the hidden state as a learnable map, I show how simple assumptions about memory updates lead directly to these complex equations. This derivation reveals that you could have arrived at the same insights by demanding that writes can be immediately read back or by taking a single step of gradient descent on a reconstruction loss.