Running a 1-Bit LLM Directly in Your Browser with WebGPU
1-Bit LLM in the Browser
I am excited to share Bonsai, a groundbreaking 1-bit Large Language Model running entirely in your browser using WebGPU. This project by the webml-community on Hugging Face demonstrates how extreme quantization allows complex AI to operate locally without heavy server costs. You can now experience real-time inference directly in your web browser, proving that powerful models don't always need massive hardware.
The future of AI is not just about bigger models, but about running them efficiently right where you need them: in your browser.
- stfurkan
I'm currently working on an open-source engine [1] exactly for this purpose. If anyone wants to try or has any suggestions, I'm happy to listen :)
1. https://github.com/stfurkan/bitgpu
2. https://aidekin.com --> this is one of my projects that's currently using bitgpu engine
- lelanthran
1-bit is not much though. Here's what I got:
Me:
> Describe the process of pasteurisation
Response:
> pasteurization is a, which a, the which which is, the and process, the the past, the and the, and and and past, and past, and and and and the, the and past, process is the is is is is, and the, past, and and future, process, and and and and and and and the, past, process, process, the the, the the, is and the, the the, and and and, and process, and and the, past, and and and past, and the, the, and the past, the the, process, process, and past, the past, past, the and, the past, and and and and and and and and and the, and and and and and and and, the the, the the, the or and and, the the, the and the, which the, and the, the the, past, and n the process, and and and and, past, and, and and, the past, and, the the, past,, and the, the the, the is the, past, and and and and and, and and and past, the and the, the the, the the, are and past, and which the, the and n, n the, the n past, past, n the, and n, the the process, which past, the the, the n, the the, the is past, the the, is, past, the the, past, and past, process, the the, the the, the and and the, and which past, the
(and that basically just goes on and on like that)
- Lwerewolf
Pretty sure this might be a duplicate. Regardless, tried the 1bit bonsai 27b gguf three different ways - their llama.cpp fork (prism, was it) on 2 machines (1255u/16g, m5 max/128g) and the web (on the 1255u). With llama.cpp it works. on the web it emits the same thing over and over. Prompt was "do you see anything wrong with this code: <C code, compound literal passed as a pointer to a function in an if statement>", result was literally "Yes, let'ssomestruct_struct_tsomestruct<a bit of similar garbage>structstructstruct<forever>.
Locally, it's definitely not the full 3.6 27b, but for ~6G with the context, it's quite impressive, and I like what this implies for larger models. Speeds on the 1255u are abysmal, though (~1TPS or less) - granted, CPU-only.
- om8
This project needs webgpu -- I did it on cpu about a year ago.
My demo uses 2 bit quantization to run llama3 models on any device with enough ram.
- totallygeeky
Impressive, I asked it how many r's in strawberry and it was immediately wrong with 2. any further questions I asked it proceeded to re-explain why there were 2 r's in strawberry, despite asking the silly "walk or drive to get my car washed" question. I struggle to see what these sorts of models could possibly be good at.