Inkling-2-Mini squeezed to 5.4 GB beats a 9.1 GB quant on math and code

Inkling-2-Mini in 5.4 GB Memory

LookAtMyQuant, a quantization compiler that measures each weight's impact on output and allocates bits under a size budget, compresses thinkingmachines/Inkling-2-Mini from 18.89 GB to 5.38 GB. At that size it outscores Unsloth's 9.14 GB UD-Q2_K_XL on GSM8K and HumanEval while running 41% smaller. An 8.87 GB version matches the BF16 original on GSM8K and cuts KL 2.34× on English/chat/code. Both run on Apple Silicon via custom Metal kernels at up to 73 tok/s.

At 5.38 GB, Inkling-2-Mini-LookAtMyQuant-5.4GB outscores Unsloth's 9.14 GB UD-Q2_K_XL on both GSM8K and HumanEval at 41% smaller size.

More from this day

2026-11-21