Why Mini PCs Run 70B Models That Big GPUs Cannot Handle
Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't
I explore how unified memory in mini PCs allows them to load massive 70B models that high-end discrete GPUs like the RTX 5090 simply cannot fit. While these compact machines offer incredible capacity for a fraction of the cost, they pay a steep price in speed due to lower memory bandwidth. The real bottleneck often isn't generation speed, but the painfully slow prompt processing times when handling long documents.
A mini PC and a used RTX 3090 are not two points on one scale. They are opposites.
- bdcravens
> Put two machines on a desk, each about $2,000. One is a tower with an NVIDIA RTX 5090: 32GB of the fastest consumer memory ever shipped, 1,792 GB/s. The other is a mini PC the size of a paperback, an AMD Ryzen AI Max+ 395 "Strix Halo" box with 128GB of soldered memory at roughly 256 GB/s.
Doesn't change the conclusions of the article, but each of those machines is more like $4k+
https://www.microcenter.com/product/711961/amd-ryzen-ai-halo...
- _davide_
I'm writing my own inference engine for Strix Halo and the same model. I already have 30%+ performance plus a more graceful decay over long contexts; that said, their point stands: memory bandwidth is what you really want.
- NortySpock
"integrated graphics processor, using system memory" had its name dragged through the mud for decades.
So we had to rebadge it to "unified memory".
Curious if we'll ever see some old integrated graphics processor "hacked" to manage to handle 128 GB of allocated system RAM and be able to serve diffusion-LLMs at a decent rate on "old" hardware...
- Havoc
Think future generations of AMD could get quite interesting. They’re no doubt seeing people whining about mem throughput specifically
- cocodill
For some reason, this reminds me of my last shared memory system. It was an Athlon XP 1800+ with VIA ProSavage back around 2002. It was just barely able to run CS 1.6.