Shoehorn - Quantize any model to fit your machine's memory
Show HN: Shoehorn – Quantize any model down to run on your machine
Shoehorn is a tool that makes any language model fit your machine's actual memory. Instead of preset quantizations that waste headroom or fail at load time, Shoehorn measures your available memory, subtracts inference overhead, and solves a per-tensor mixed-precision assignment to use up to 99.99% of your budget. It integrates with llama.cpp and offers a local web app where you can discover models ranked by quality, fit them with one button, and chat. Available via Homebrew or Cargo.
Shoehorn starts from the memory you actually have, subtracts what inference itself needs, and solves a per-tensor mixed-precision assignment that lands within a rounding error of the remainder — routinely using 99.99% of the budget, sometimes to the byte.
- hmokiguess
Reminds me of https://github.com/AlexsJones/llmfit
- mbuchel-hn
does this work similar to airllm? i am wondering how it would handle something like quantizing kimi k3 on a budget of 8 gbs, or is that something you are not attempting to solve yet?
- jaylane
tried it out but based on the model sizing result i got i got an insufficient memory error when the server started running