quantization

Reducing the numerical precision of model weights and activations to shrink memory footprint and speed up inference, covering methods like GPTQ, AWQ, GGUF, and int8 or 4-bit schemes. Content here digs into the real tradeoffs between size, latency, and accuracy, including how aggressively you can compress before output quality collapses. Expect hands-on findings on where standard quantization advice breaks down, especially for small models that behave differently than the large ones.

quantization, local-llms, small-models, ai-development

Ninety-Seven to Two: What Quantization Does to a Small Model

We shipped a 1.5B router that produced valid JSON 97% of the time. One quantization step took it to two. Here is the day we spent finding out why — and why the rules everyone repeats about quantization were written for models ten times the size. It started with a