Count the Bytes, Not the FLOPs: Why We Stopped Materializing Arrays

Count the Bytes, Not the FLOPs

We discovered our bootstrap engine was slower than competitors because we prioritized FLOPs over memory traffic. By stopping the materialization of giant tensors and eliminating unnecessary state, we reduced memory usage from gigabytes to megabytes. This shift from batching everything to streaming data through cache delivered performance gains of up to 20 times, proving that moving fewer bytes often beats raw computation.

"The gather is slow not because the code is interpreted but because its arithmetic intensity is far too low to keep the processor busy: it is a bytes problem being graded as a FLOPs problem."