Hot Chips 2026: High Bandwidth Flash Could Slash DRAM Costs for AI, but Software Hurdles Loom
Hot Chips 2026: Applying High Bandwidth Flash (HBF)
At Hot Chips 2026, a tutorial explored High Bandwidth Flash (HBF), a new memory technology that packages flash memory like HBM on the same chip as a processor. HBF offers far greater capacity than HBM but with lower bandwidth and block-based access, requiring software to treat it like a storage device. The talk examined how HBF could be applied to machine learning, including storing MoE experts and KV caches in flash, and replicating model weights to reduce cross-device communication. However, the software changes needed are substantial, and HBF's cost-effectiveness depends on workloads that aren't bandwidth-bound.
Taking advantage of SSD actually seems easier. The OS kernel can abstract away the difficulty of doing block-aligned accesses if you don’t use FILE_FLAG_NO_BUFFERING or O_DIRECT.