Twelve Thousand Prompts and the Uncomfortable Truth About What Developers Actually Ask

We set out to teach a small model how to route developer prompts. The data taught us that our taxonomy — and our sources — encoded a picture of developers that doesn't exist.

It started with a quota.

I wanted a small, fast model that could sit in front of a coding assistant and do one job well: take a raw developer prompt, clean it up, classify what kind of request it is, and decide which downstream model should answer it. A router, not a chatbot. The kind of short, structured, high-volume task where a local specialist actually beats a frontier API.

To train it, I'd distill a bigger model: feed thousands of real developer prompts to a strong teacher model, have it produce the ideal structured output for each, then use a second, cheaper model as a judge to keep only the good ones. Standard distillation. The interesting decision was the taxonomy — I'd defined ten intents (debug, feature, explain, refactor, architecture, optimize, review, boilerplate, documentation, and a catch-all "other") and set a target count for each so the dataset would be nicely balanced.

That balance target is where the trouble started. Or rather, where it surfaced — because the trouble was already baked into the data, and I just couldn't see it yet.

The first lie: labels don't mean what they say

Two of my ten classes refused to fill up. "Review" and "boilerplate" came in far below target no matter how many seeds I threw at them.

My first instinct was a supply problem: I just needed more review and boilerplate prompts. I had thousands of GitHub issues tagged with labels like "Suggestion," "Proposal," and "In Discussion" — surely those were code-review and design requests. I pulled 500 of them and ran them through the teacher.

The teacher classified 5 of those 500 as "review."

The other 495 came back as feature requests, debugging, or architecture discussions. And the teacher was right. A GitHub issue labeled "Suggestion" is somebody proposing a feature, not somebody asking "review my code." The label described how a maintainer triages an issue tracker — it had almost nothing to do with what the prompt was asking for. Across the board, the teacher reclassified roughly 96% of my "review" and "boilerplate" seeds into other intents.

This was the first crack. The labels I'd trusted to seed my categories were a different vocabulary spoken by a different audience for a different purpose. The word "review" on a GitHub issue and the word "review" in a developer's prompt to an assistant are false friends.

If you take one thing from this: the label on your source data is not the label you think it is. When a strong content classifier disagrees with your scraped labels, bet on the classifier.

The distribution trap: your dataset is your source

That discovery sent me to do something I should have done at the start — measure the intent distribution per source instead of in aggregate. My seeds came mostly from GitHub issues, with smaller amounts from StackOverflow and a public corpus of real developer-to-chatbot conversations.

Here's what the teacher's own classifications looked like, broken out by where the prompt came from:

Look at the debug row. In GitHub issues, 42% of prompts are debugging. In actual developer-to-assistant conversations, it's 11%. Look at explain: 6% in issues, but 26% in real chat logs and 40% on StackOverflow. People ask assistants to explain things constantly; they almost never open a GitHub issue to do it.

My dataset was 90% GitHub issues. Which means my "balanced" dataset wasn't a picture of what developers ask coding assistants. It was a picture of what people file in issue trackers — a genre dominated by bug reports and feature proposals, and almost entirely missing the "explain this to me" and "scaffold me a thing" requests that make up a huge share of real assistant traffic.

I had been about to ship a router whose entire worldview was "everything is a bug or a feature," because that's what issue trackers are made of. The model would have learned to over-predict debug on every ambiguous prompt — and it would have been faithfully reproducing a bias I introduced by where I scraped, not anything about how developers behave.

The intent distribution you measure is the distribution of your source, not of your users. You cannot see this in the aggregate numbers. You can only see it when you cut the same classifier across different sources and watch the rows swing by 4x.

The goldmine in the reject pile

There was a tenth category I'd been treating as garbage: "other." Anything the teacher couldn't fit into the nine real intents got dumped there and filtered out before training.

On a hunch, I clustered the 400-odd "other" prompts that had passed the quality judge and read them. About a quarter of them were the same thing: requests to write or fix documentation — clarify a docstring, correct an API reference, improve a README. Not debugging (nothing was broken), not explaining (the user didn't want to understand it themselves), not a feature (no new code). A genuine, coherent category I simply hadn't included at first in my taxonomy.

So I added it. "Documentation" became the tenth intent, the teacher prompt learned to recognize it, and I recovered about 250 high-quality examples that had been heading for the trash.

The lesson here is cheap and universal: audit your reject pile. The "miscellaneous" bucket isn't noise — it's the shape of the categories you forgot to define. If a quarter of your rejects are secretly the same thing, that's a missing class, not garbage.

When the right move is to fabricate

That left boilerplate — the genuinely empty class. Not mislabeled, not hiding in the reject pile. Just absent. And once I understood the distribution trap, I knew why: developers don't open GitHub issues that say "scaffold me a CRUD endpoint." They do that inside their editor, or in a throwaway chat. Boilerplate is a real and common request to an assistant, and structurally invisible to my sources.

You can't scrape your way out of a class that your sources don't contain. So I did the thing that felt like cheating: I generated boilerplate prompts synthetically — varied templates across endpoints, Dockerfiles, configs, data models, CLIs, test stubs, UI components, with the language and framework randomized.

The teacher classified 90% of the synthetic prompts as boilerplate — versus the ~3% acceptance rate of the "boilerplate"-labeled GitHub seeds. That gap is the whole story in one number. Clean, unambiguous, purpose-built prompts read as boilerplate because they are boilerplate. The scraped ones never were.

Synthetic data has a bad reputation, and often deservedly. But for a class that is formulaic by nature and absent from your real sources, fabricating it is not a workaround — it's the correct tool. The trick is knowing which classes qualify (formulaic, low-ambiguity) and which would be poisoned by it (anything where the messiness of real prompts is the point).

The cheap trick that paid for itself

One practical note, because money matters. The judge model — the one scoring every teacher output — was the expensive part of the loop. So I added early-stopping by quota: once a class hit its target count of accepted examples, stop sending its candidates to the judge entirely. Run the cheap teacher to classify, and if the class is already full, skip the costly judgment.

On the full run that skipped over seven thousand judge calls and cut roughly thirty dollars off a single pass — a meaningful fraction of the bill, for a five-line change. When your pipeline has a cheap stage and an expensive stage, let the cheap stage decide when the expensive one is even worth running.

The honest caveat

I owe you the error bars. The most damning numbers in this piece — the per-source distribution swings — lean on the smallest of my sources. The real developer-to-chatbot sample was only a couple hundred prompts. The direction of the effect is too large to be noise (a 4x swing in debug isn't a sampling artifact), but the exact percentages should be read as "roughly this," not gospel. Before I trust these numbers to set a final distribution, I want a bigger real-traffic sample to confirm them. Calling a 26% out loud when it might be 20% or 32% would be exactly the kind of overconfidence this whole post is warning against.

The uncomfortable truth

After weeks of building collectors, tuning prompts, balancing quotas, and judging outputs, the single most valuable thing the project produced wasn't the dataset. It was the discovery that I didn't know what developers ask — and neither did my data.

Every dataset is a measurement instrument, and like any instrument it has a bias you can't read off its own dial. My taxonomy encoded my assumptions about how developers think in categories. My sources encoded the genre conventions of wherever I happened to scrape. Both looked perfectly reasonable in aggregate. Both were quietly wrong, and they were only exposed when a model strong enough to actually read the prompts disagreed with the labels I'd stapled onto them.

The uncomfortable part isn't that the data was biased. All data is biased. The uncomfortable part is how invisible it was. The aggregate distribution looked balanced. The labels looked authoritative. The reject pile looked like garbage. Nothing about the dataset, viewed from the top, told me it was a portrait of issue-tracker culture wearing a developer's name tag.

If you're building a specialist model from distilled or scraped data, don't ask "is my dataset big enough?" Ask "whose behavior is this actually a recording of?" Cut your classifier across every source you have and watch the rows move. Read your reject pile. Trust the model's read of the content over the label on the file. And when a class is genuinely missing because your sources never contained it, build it on purpose instead of pretending a mislabeled proxy will do.

The models are good enough to teach a smaller model almost anything. The hard part was never the teaching. It was figuring out that I'd been about to teach it a confident, well-balanced, completely fictional version of my own users.

André builds and breaks ML systems at AIpster, an independent AI think tank based in São Paulo, Brazil. This piece came out of building a prompt-routing dataset by distillation — and discovering the dataset had opinions about developers that the developers didn't share.