DRAM Is About to Do Math: Samsung's LPDDR5X-PIM Delivers 8x Bandwidth
Processing in Memory: DRAM Is About to Do Math
At Hot Chips 2026, Samsung unveiled a 16 GB LPDDR5X package with processing-in-memory (PIM) that delivers 614 GB/s of internal bandwidth—eight times what escapes through its pins. This matches the memory bandwidth of a top-spec Apple M5 Max, but from a single memory package. The design places compute units next to DRAM banks, enabling 3x faster AI inference on Llama 3.1 8B compared to standard LPDDR5X. However, software challenges remain: k-quants don't map to the hardware, physical memory layout requires bank-aware allocation, and the design favors GEMV over GEMM, limiting compatibility with modern inference techniques like speculative decoding and mixture-of-experts.
DRAM banks already provide most of the bandwidth; the external pins cannot expose it.
- roadbuster
"Processor-in-Memory", is an age-old idea which was even explored in a doctoral thesis from 1998: "Computational RAM: A Memory-SIMD Hybrid"*. At the time, the motivation was, "Between a DRAM chip and a microprocessor mounted on a circuit board, we can only get a few hundred megabytes per second of bandwidth. However, inside the memory chip itself, there's hundreds of gigabytes (terabits) of bandwidth available, so why not move some of the logic there?". And then the author pushed it to the extreme where there was a wide array of 1-bit processors at each column of DRAM. (Note: the reason why true PIM isn't viable is because of the radically different semiconductor process used for DRAM vs. high-speed digital logic, i.e.: your digital logic circuits will perform poorly on a "DRAM process").
The designers of HBM5 had the same observations and thought, "why don't we put a logic chip at the bottom of the stack of HBM DRAM chips and get extremely high memory bandwidth?" But, in looking at this, you'll also quickly realize this is almost exactly what Intel and Micron pitched 15 years ago as Hybrid Memory Cube** (which lost on the market compared to the simpler, less costly, and more flexible alternative, HBM)
* https://www.eecg.toronto.edu/~stumm/Theses/Elliott-PhD98.pdf
- honr
Compute In Memory has been most of what electronics research (more accurately, the parts I am interacting with) talking about this year. There are a few trends and I am curious to see which ones survive in the market. RAM designers are hot in the market and job openings at the moment.
- MBCook
So they built a very specific operation into the RAM. This RAM would only be useful for AI, otherwise the transistors are just wasted.
And if we fund a different set of operations actually work better for AI you can’t adjust to that either.
Seems extremely limited. But otherwise wouldn’t you just end up with something like a massive number of tiny microcontrollers each accessing a set of memory? The Transputer again.
- jcranmer
Processing in memory was one of the extra topics we covered in my grad computer architecture class. Fifteen years ago. And even then, it was very clearly an old-hat idea.
One of the problems with trying to move a lot of ALUs close to the DRAM logic is that ALUs are pretty hot, and DRAM is some of the most sensitive components to heat. There are some other issues too, like the process manufacturing differences mentioned by a sibling comment.
- senshan
Why does not the article say anything on how KV-cache is handled with in-DRAM processing? Is this trivially obvious?