Run Massive Llama 3.1 and Falcon Models at Home BitTorrent-Style
Run large language models at home, BitTorrent‑style
I can now run massive language models like Llama 3.1 and Falcon on a consumer-grade GPU by joining a distributed network. Instead of downloading the entire model, I load just a part while others serve the rest, enabling interactive chatbots and fine-tuning. This approach combines the ease of an API with the full flexibility of PyTorch and Transformers, all powered by the BigScience research workshop.
You get the comforts of an API with the flexibility of PyTorch and Transformers.
- woctordho
Petals is from 2022. Nowadays intelligence of smaller models, quantization techs, and optimizations to run models faster on consumer GPUs have improved a lot.
For distributed inference of smaller LLMs and diffusion models that fits in one consumer GPU rather than splits on multiple machines, there are already pretty good solutions such as AI Horde (formerly Stable Horde) [0]. Notably, it's the default provider that powers SillyTavern. It also has an interesting economy model of kudos.
- N_Lens
It's an interesting concept but the timing is probably too early. If more people had reliable low latency gigabit or ideally 10gbit throughput, then it might start to approach feasibility. There's other blockers too, but that springs to mind immediately.
It's cool to imagine a planet wide neural network interconnected with fiber - the nervous system of a planetary intelligence. But perhaps mushrooms do that already (Alpha Centauri ever relevant).
- maccam912
The recently discussed https://meshllm.cloud/ is the one I've been playing with but don't have the hardware to try with a model split between nodes, which apparently is supported and just not part of the public demo.
- novia
The reason i want to run models at home is to be able to experiment on them, change things, see how they get better or worse.
- wxw
Hm, looks like this is an old project (2022) associated with HuggingFace (https://huggingface.co/bigscience).
Doesn't look like it's very active nowadays: https://github.com/bigscience-workshop/petals
Article with more details: https://techcrunch.com/2022/12/20/petals-is-creating-a-free-...
> [...] volunteers can donate their hardware power to tackle a portion of a text-generating workload and team up others to complete larger tasks, similar to Folding@home and other distributed compute setups.