FlashAttention: The Exact Attention Algorithm That Cuts GPU Memory Traffic, Not Math
Understanding FlashAttention Pt 1: Personal Notes
A technical handbook on FlashAttention explains how tiling, online softmax, and recomputation make attention IO-aware without approximating the math. It walks through GPU memory hierarchy, the quadratic cost of materializing score matrices, and the evolution from FlashAttention-1 to -4. The core insight: wall-clock speed depends on data movement, not just FLOPs. Exactness means the same function, not bitwise reproducibility.
A computational graph is not a memory schedule.