Mesh LLM: Run Distributed AI on Your Own Hardware with iroh
Mesh LLM: distributed AI computing on iroh
I built Mesh LLM to stop teams from surrendering control and paying endless bills to big AI providers. By pooling GPUs across your own devices using iroh, we create a private mesh that runs massive models locally or splits them across machines. You get a standard OpenAI-compatible API at localhost while keeping your data secure and your costs low.
The popular models are monoliths, and reaching them through an API key is a surrender of control over your data, hardware, and costs.
- maccam912
I have a macbook pro, figured I'd see how easy it was to contribute some vram...
And I can't overstate how easy it was. The swarm page thing had a little "join" button and said to run "mesh-llm --auto". And I did. And it worked first try. That is such an uncommon experience I had to report back. It handled picking a model to serve, downloading it from peers, and to test it I chatted with the model I was hosting, I could see the GPU doing work, etc.
It might be more of an endorsement for iroh than mesh-llm, although I'm sure getting it to all work seamlessly took work on both sides. But to whoever spent the time and energy trying to make it seamless, consider the effort recognized!
- SwellJoe
I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks.
Are we talking 1 token per second for a split model? Less?
Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.
- MattPerry
The first picture "gpu rig", "laptop", "server", "cloud node, etc made me realize how little compute I have. I don't have a laptop with 24GB VRAM or a workstation with 96GB. I think if I convinced all of my friends to run LLMs on their gaming PCs, I don't I would have the total VRAM in the picture.
As an aside, I saw this post mentions a public mesh, but I couldn't find any more information.
- i386
I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
- Abishek_Muthian
I'm more interested in running distributed inference for purpose built small language models than these coding LLMs.
Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output.
Nicely done OP.