Apple Caught Off Guard by AI Demand for Mac Mini and Mac Studio

I’m curious to know if these local AI setups are legitimately useful compared to cloud. I’ve struggled a lot to get something useful out of the hardware I have.

I realize I’m somewhat limited (16GB RX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me.

Any tips anyone might have are appreciated! I’d love to be local first and would be willing to buy hardware to get there.

It's fun to see that even an extremely large company can find unexpected product market fit [0]. Per this article, "The company reportedly did not possess an engineering team dedicated to business customers or staff focused on developer relations, and lacked an enterprise AI strategy." That sounds insane in retrospect, but I think there's just inherent uncertainty in what people actually need and will use things for.

[0]https://pmarchive.com/guide_to_startups_part4.html: "In a great market—a market with lots of real potential customers—the market pulls product out of the startup... The product doesn’t need to be great; it just has to basically work."

Mac Mini's were really nice HTPC candidates, too, before the AI boom. Like all things genuinely useful and affordable, they were snatched from the hands of normal consumers by a bunch of schmucks chasing the latest gold rush.

No they weren’t.

I really hope with Ternus taking the helm Apple starts to remember that it has products outside of iPhone.

Not just the high end stuff. The Neo is sold out until late September on the budget end, it seems like it is a smash for HS and college kids.

I hope Apple can take all this cash and do some stability releases like they used to do, bugs around things like Family Sharing, the painful "update" to Settings App, etc could all use a lot of love.

So for people who don't understand, there are two markets for Apple hardware in this space:

1. Running an agent like OpenClaude. The $599 Mac Mini was an insanely good deal for this. I happened to buy a M5 Pro Mac Mini for $999 last year for other reasons. The equivalent is now almost $2000; and

2. Hardware for running inference on local models. This to me is the far more interesting market because Apple has a real opportunity to disrupt NVidia's stranglehold on the market.

With current architecture, the largest model you can reasonbly run is the amount of memory on the GPU and is a function of the quantization (eg int4, int8, fp8, fp16, etc) available and the number of parameters. NVidia aggressively segments the market. The most VRAM on a "consumer" card is 32GB on the 5090, which allows you to run ~31B parameter models.

In comparison, the RTX 6000 Pro has only slightly more CUDA units than a 5090 but has 80GB of VRAM. A few months ago they were $10-11k. Now they're ~$16k.

Macs use a shared memory architecture. Apple has previously sold Mac Studios with up to 512GB of RAM. Almost all of that memory can be used to hold much larger models without taking a penalty for interconnections between different GPUs or machines. Plus Apple interconnects between computers are actually relatively good by chaining TB5. It's still slow but it's about the best non-enterprise option available.

But the previous Mac Studios just didn't have the raw FLOPS and memory bandwidth. The M5 Ultras are […]