Ninety-Seven to Two: What Quantization Does to a Small Model

We shipped a 1.5B router that produced valid JSON 97% of the time. One quantization step took it to two. Here is the day we spent finding out why — and why the rules everyone repeats about quantization were written for models ten times the size.

It started with a model that worked.

We had spent days building DevRouter-1.5B, a tiny specialist that reads a raw developer prompt and returns a single JSON object: a cleaned-up rewrite, an intent label, a complexity guess, a routing decision, and a list of the context the prompt forgot to include. The whole point is to sit in front of your expensive models and make a cheap triage call before you spend real tokens.

By the time we finished evaluating it, the numbers were good enough to stop. On a held-out validation set it produced strictly valid JSON 97.3% of the time, classified intent at 0.71, route at 0.74. On an out-of-distribution set — prompts from sources it never trained on — validity held at 95.5%. For a 1.5-billion-parameter model doing structured output, that is a model you ship.

So we packaged it. And that is where the story actually begins.

Two percent

To ship a model people can actually run, you convert it to GGUF — the format llama.cpp and Ollama speak — and you quantize it, trading a little precision for a much smaller file. The community consensus is comfortable and well-worn: Q4 is fine for most things, Q5 and Q6 are basically free, only the desperate go below Q4. We picked Q6_K. Near-lossless, everyone says. We had a working model; we just wanted a smaller copy of it.

We ran the same evaluation against the quantized file.

JSON validity: two percent.

Not a regression. A collapse. Ninety-seven to two. The model that had been a reliable little JSON machine an hour earlier was now failing to produce parseable output on 98 of every 100 prompts.

The strange part was what the failures looked like. The model had not turned to noise. It was still coherent, still on-task, still clearly trying to route the prompt. It just couldn’t keep the JSON together. It would write "complexity": medium — the right answer, no quotes. It would put a free-text sentence where an enum value belonged. It would start the missing field as a bare string instead of an array. The intelligence was intact. The discipline was gone.

That distinction matters, and it is what made the next few hours interesting.

The red herring

When a model serves fine in one engine and breaks in another, the first suspect is never the weights. It is the plumbing.

So we went after the plumbing. We were serving the GGUF with llama.cpp’s server, and a quick read of the logs turned up something real: the server had quietly set itself to four parallel slots and split our context window across them. We had given it 4096 tokens; each slot got 1024. Our prompts plus the model’s long rewrites routinely blew past that, so the output was being truncated mid-object. There was our broken JSON.

We fixed it — one slot, full context — and felt clever for about ninety seconds. Then we re-ran it.

Still two percent.

So we kept pulling. Maybe it was the chat template the GGUF shipped with, subtly different from what the model trained on. We bypassed the template entirely and fed the model a hand-built prompt, byte-for-byte like training. Still broken. Maybe it was the sampler — llama.cpp applies a repetition penalty by default, and JSON is nothing but repeated punctuation. We forced pure greedy decoding, penalties off. Still broken.

Every comfortable explanation failed. The plumbing was fine. We had run out of places to hide.

The ladder

When you’ve eliminated the easy answers, you stop guessing and start bisecting. We had four versions of the same model, and we could test each one in isolation:

• The LoRA adapter on its quantized training base, run through the training framework: 97% valid.
• The adapter merged into full 16-bit weights, run through plain Transformers: 100% valid on the hardest, longest examples in the set.
• The Q6_K GGUF, run through llama.cpp with every confound removed: two percent.
• A freshly converted, un-quantized F16 GGUF — same conversion path, same server, same prompt: valid.

Read that ladder twice. The merge is clean — 100%. The conversion is clean — F16 works. The serving is clean — F16 works through the exact same server that mangles Q6. The only thing that changes between the model that works and the model that doesn’t is six-and-a-half bits per weight instead of sixteen.

It was the quantization. Just the quantization. The thing everyone calls free.

Why the big-model rules don’t apply

Here is the part that reframed it for us. The folklore isn’t wrong, exactly. Q6 is nearly lossless — on a 70B. The rules of thumb the local-LLM world repeats were measured on large models, and on large models they hold up beautifully, because a large model is mostly redundancy. Knock a little precision off any given weight and there are a thousand other paths carrying the same signal.

A 1.5B model has no such luxury. There is less redundancy, less slack, fewer second chances. Quantization sensitivity scales inversely with size: the smaller the model, the more a fixed amount of precision loss actually costs you.

And it doesn’t cost you uniformly. It comes for your most brittle behavior first. Open-ended chat degrades gracefully — a slightly worse word here, a blander sentence there, nothing a human notices. Strict structured output has no graceful degradation. There is no “almost valid” JSON. A single dropped quote or a comma in the wrong place flips the entire output from usable to garbage. The model still knows the answer is "medium"; quantization just nudged the probability of emitting that closing quote below the probability of moving on, and the brace never closes.

So you get exactly what we got: a model that is still smart and no longer parseable. The capability survives. The format collapses. And format was the entire product.

Eight bits, and a joke about 400 megabytes

The fix was boring, which is how you know it’s right. We quantized to Q8_0 instead — eight bits, much closer to the original — and re-ran the full evaluation. Validity came back to 96.5% in-distribution and 94.6% out-of-distribution, with intent, route, and complexity all within a few points of the unquantized model. Q8 holds. We shipped it, with an F16 build alongside for anyone who wants zero compromise.

Now the joke. On a 1.5B model, Q6_K is about 1.2 GB. Q8_0 is about 1.6 GB. F16 is about 3 GB. The entire reason you reach below Q8 is to save space — and here the difference between “broken” and “works” was four hundred megabytes. We almost destroyed the one thing the model was built to do in order to save a rounding error of disk.

That is the trade nobody puts on the chart. For a large model, dropping a quant level saves you tens of gigabytes and you’d be foolish not to. For a small one, you’re risking the whole model to save the size of a phone photo album.

What we’re not claiming

This is one model and one task, so let me draw the line carefully. We are not saying Q6_K is broken in general — it is excellent on larger models, and it may well be fine on a different 1.5B doing a less brittle job. We are not saying our numbers transfer to your setup; single-stream throughput, a specific inference engine, and a strict-JSON objective all shaped what we saw. Some of the gap between the Transformers run and the llama.cpp run is the two engines disagreeing at the last decimal of “greedy,” not quantization alone.

The claim is narrower and, we think, more useful: on a small model doing strict structured output, do not assume a quant level is safe because the internet says so. Measure it. And measure it the right way — by serving the actual quantized file and running your real evaluation, not by eyeballing one short prompt. Every short prompt we tried passed. The breakage only showed up on long outputs, which is exactly where a one-line smoke test will lie to you and a real test won’t.

Quantize like your model is small

The small-specialist-model movement is the part of this field we’re most excited about — the idea that you don’t need a frontier model to do a narrow job well, that a 1.5B can sit in your pipeline and earn its keep. But that excitement comes with a tax that the large-model playbook hides. The defaults the community hands you — the quant levels, the “Q4 is fine,” the near-lossless reassurances — were written by and for people running giant models, where they are simply true.

Apply them unexamined to a tiny model and you can spend a day, as we did, watching a model you carefully built fail in a way that looks like a bug in everything except the one thing that’s actually wrong. The smaller the model, the less you can borrow someone else’s confidence about precision. You earn that confidence per model, with a real eval, or you don’t have it.

Ship Q8. Validate before you trust the file. And when a working model breaks the moment you shrink it, climb the ladder one rung at a time — the answer is usually sitting on a rung everyone told you was safe to skip.

DevRouter-1.5B is open and Apache 2.0:

• aipster/DevRouter-1.5B — fp16 weights (Transformers / vLLM)
• aipster/DevRouter-1.5B-GGUF — Q8_0 + F16, plug-n-play with Ollama and llama.cpp

ollama run hf.co/aipster/DevRouter-1.5B-GGUF:Q8_0 "refactor this giant function into smaller ones"

We build to understand. We share to learn together — including the day we almost shipped two percent.