AirLLM Runs 70B LLMs on a Single 4GB GPU
AirLLM 70B inference with single 4GB GPU
AirLLM slashes inference memory usage, enabling 70B models to run on a single 4GB GPU without quantization, distillation, or pruning. It streams one layer at a time, so VRAM needs depend on layer size, not total model size. This lets you run Llama 3.1 405B on 8GB, DeepSeek-V3 671B on ~12GB, and even Kimi K3 2.8T on under 4GB. The library supports virtually every popular open LLM via AutoModel, with optional 4-bit/8-bit compression for up to 3x speedup.
AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning.
- imenani
For anyone wondering “how slow is this?”
IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token
- roger_
Seeing a lot of these “run 1TB models with 1GB RAM” projects recently. Most seem vibe coded and probably won’t be maintained.
Hoping a winner emerges with some real momentum behind it.
- seu
I love it how the rampocalypse is pushing people to squash all the performance they can. I hope that this also leads to rethinking model architecture so that they can do the same with less.
- xg15
You can run any frontier model on your PC if you just wait long enough...
- cpfohl
I’m still slightly confused on what this adds.
Let’s say I wanted to run a full size open weight model. I have a 128GB m3 max laptop.
Does this basically load layers in and out on demand? So I still have to download the full model to disk, but the RAM requirements go way down? The readme calls out that one still needs to connect HuggingFace, which leads me to believe that maybe you don’t even need to download the full model?