How You Could Have Invented Kimi Delta Attention from First Principles
You Could Have Come Up with Kimi Delta Attention
I walk through the evolution from standard softmax attention to linear variants like DeltaNet and Kimi Delta Attention. By treating the hidden state as a learnable map, I show how simple assumptions about memory updates lead directly to these complex equations. This derivation reveals that you could have arrived at the same insights by demanding that writes can be immediately read back or by taking a single step of gradient descent on a reconstruction loss.
Linear attention has given us a compact associative memory, but its update behaves like += when what we want is closer to =.
- TrackerFF
Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation.
Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even.
EDIT: Didn't even notice the notation switch, much appreciated.
- benjiro29
> You Could Have Come Up with ...
Creating or combining to have something new, that does not already exist is actually freaking hard!
The moment its presented and people go "o, that is not that difficult", "i was able to also do that", or some nonsense like that. Everything looks simply the moment somebody did the hard work.
We have all been there was developers. Thinking we invented something new, and ... then you discover somebody already made it in the 70's and its everywhere. But because it never cross your path, you never realized it existed.
- dr_kretyn
The bra-ket notation makes this all very simple/intuitive for me. With "vectors" I always get confused which is horizontal/vertical, and then I just follow blobs, and get distracted, and leave. With bra-kets the whole thing was very intuitive! I'm now going to covert other articles to the notation as I must have missed a lot of good stuff!
(Side notes: I have physics PhD and mild dyslexia)
- croemer
LLM written for sure:
> The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
- HonshinM
A visualized tutorial: https://snowchord.com/blog/linear-attention-visualized/
- alex-moon
"Machine learning architecture is second nature to us AI researchers, so it's easy to forget that the average person probably only knows the formulas for gradient descent and ReLU."
"And softmax of course."
- xp84
When I see these types of articles and headlines, it just makes me supremely grateful for all the many people far smarter[1] than me. And humbles me, too, since I actually passed for a "very smart person" in places like high school and undergrad. In fact, I'm 'smart' for an average person, but there are definitely millions of people who make me look like a rube in comparison.
[1] I specifically mean those who are able to hold very big complex ideas and systems in their head, and reason about them, which seems to be an important talent for mathematicians.
- _Microft
Side note, before you ask: yes, bra-ket notation is called like that because of the brackets.