High-Bandwidth Flash Could Solve AI's Massive Storage Problem
High-Bandwidth Flash offers efficient storage for model weights
I explore how High-Bandwidth Flash (HBF) transforms standard flash memory into a powerful solution for AI inference. By stacking NAND chips, companies like Sandisk and SK Hynix aim to deliver massive storage capacity for frozen model weights. This approach frees up expensive HBM for high-speed tasks, offering a cost-effective way to scale Large Language Models without sacrificing performance.
Flash is atrociously slow for writes, but for reads, it can be coaxed to go pretty fast.
- hankbond
Necessity being the mother of all invention. I also thought the compute-in-memory approach was interesting re: https://mythic.ai
- jmward01
I didn't see cost. If cost is similar to flash drives then this could be massive. Every GPU ships with a lot of ram still but for AI inference and games you have 10TB of this stuff to stuff all your textures into and all your model weights in. For fine tuning models this would also be great if using lora or similar.
- dlcarrier
Intel really missed out, when they discontinued the Optane line, right before the RAMpocalypse.