How Transformers Store a Million Memories Without Forgetting

You could have invented transformers from memory

Large language models recall facts from across the world, but how much can they actually remember? By viewing linear layers, MLPs, and self-attention as memory stores, we can derive their capacities. Starting with a simple outer-product store that holds about 64 memories, we improve to an optimal linear store with twice the capacity, then to MLPs and self-attention, which can store over a million memories with 64-dimensional keys.