small-models

Coverage of compact language models in the 0.5B to 7B range, where the tradeoffs differ sharply from frontier-scale systems. Expect hands-on accounts of fine-tuning specialists, quantization behavior, structured output reliability, and the failure modes that only show up at small scale. The focus is practical: getting tiny models to do narrow jobs well, and questioning the conventional wisdom that was written for models ten times larger.

quantization, local-llms, small-models, ai-development

Ninety-Seven to Two: What Quantization Does to a Small Model

We shipped a 1.5B router that produced valid JSON 97% of the time. One quantization step took it to two. Here is the day we spent finding out why — and why the rules everyone repeats about quantization were written for models ten times the size. It started with a