WebLLM runs LLMs directly in your browser with WebGPU acceleration
WebLLM: high-performance in-browser LLM inference engine
WebLLM is a high-performance inference engine that brings large language models to web browsers via WebGPU, eliminating server-side processing. It offers full OpenAI API compatibility for streaming, JSON-mode, and function calling, and supports models like Llama 3, Phi 3, Gemma, Mistral, and Qwen. Developers can integrate it via npm or CDN, and use workers or service workers to keep the UI responsive.
Everything runs inside the browser with no server support and is accelerated with WebGPU.
- TekMol
This seems to be the demo:
I am getting:
WebGPUNotAvailableError: WebGPU is not supported in
your current environment, but it is necessary to
run the WebLLM engine.
On both, FireFox and Chromium on Linux.
- MarioMan
I really enjoy this engine. I’ve used it for personal projects, but it hasn’t been updated since Gemma 2. I suggest using Transformers.js instead these days.
- init0
You might like webml-kit https://npm.im/webml-kit