What Actually Gets Distilled: Copying the Method Is Not Copying the Judgment

TL;DR. Knowledge distillation reliably transfers a model's surface behavior, its outputs, style, and benchmark scores, while losing the properties that made that behavior trustworthy: calibration, robustness, and faithful reasoning. The same split applies to human expertise absorbed by AI. Your explicit method transfers cleanly because it was already writable.

Part 2 of a three-part series on what happens to expertise once it goes public in the AI era.

A distilled model may closely mimic what another model does—producing similar answers, adopting the same tone, and achieving comparable test results—without inheriting the deeper qualities that made those results dependable. Its confidence may be less accurate, its performance may break under pressure, and its reasoning may no longer track the process it appears to follow. A similar limitation arises when AI learns from human experts. Procedures and frameworks are easy to capture because they can be stated explicitly. Instinct, context-sensitive judgment, and experience-based discernment are much harder to reproduce—not because experts are concealing them, but because much of that knowledge was never converted into language.

The instinct from Part 1, made precise

Part 1 named the paradox: publishing real expertise builds your authority while feeding the systems that commoditize it. Most people, when they feel that tension, land on a simple defensive instinct. Don't show the method. The method is the crown jewel, and once it's public, anyone (or any model) can copy it.

That instinct is half right and half backwards. This piece asks the sharper question: commoditize what, exactly?

The research on model distillation gives a precise, transferable answer. It turns out the thing that copies easily and the thing that stays yours are not what the defensive instinct assumes. The method is the part that copies. The judgment behind it mostly doesn't. Understanding why requires a short detour through how one model learns to imitate another.

The retention assumption: matching the output is not matching the capability

When engineers want a smaller, cheaper model, they often build it through distillation. A large "teacher" model produces outputs, and a smaller "student" model is trained to imitate them. The student learns to sound like the teacher, answer like the teacher, and, ideally, score like the teacher on benchmarks.

Here's the quiet logical leap in how we usually judge that process. We measure whether the student matched the teacher's headline score, then conclude the student inherited the teacher's underlying capability. A 2026 position paper calls this the retention assumption and argues it's exactly the thing distillation evaluation keeps conflating (Wang, "Knowledge Distillation Must Account for What It Loses," 2026, arXiv:2604.25110).

The paper synthesizes prior evidence that students can match a teacher's benchmark while diverging on the properties that made the teacher reliable: calibration (knowing how confident to be), robustness (holding up under pressure), privacy behavior, and the faithfulness of reasoning traces (whether the stated reasoning actually produced the answer, or just sounds like it did).

Read that list again, because it's the whole argument in miniature. A model can produce the right answer for the wrong internal reasons, and standard scoring will never notice. The output looks identical. The thing that generated it is not.

The gap between "looks the same" and "is the same" is large and mostly invisible

This could be a philosophical worry. It isn't. It shows up as hard numbers even in narrow, well-behaved domains.

Consider code generation, about as objective a task as machine learning offers. Code either runs or it doesn't. A 2025 metamorphic-testing study distilled code-generation models and then stress-tested them with behavior-preserving transformations, small rewrites that shouldn't change whether the code works (Awal, Rochan and Roy, 2025, arXiv:2511.05476).

The distilled students held their conventional accuracy scores. On paper, they matched. Under the transformations, they showed up to a 285% greater performance drop than their teacher. The researchers found behavioral discrepancies in up to 62% of cases that ordinary accuracy-based evaluation missed entirely.

Sit with those two numbers together. Standard evaluation said the student was a faithful copy. The student was nearly three times more fragile under pressure, and the usual tests couldn't see it in roughly six out of ten cases.

If the gap between imitation and capability is that wide in code, a domain with a compiler as ground truth, it is wider still in domains where correctness is a judgment call: a legal strategy, a clinical read, a pricing decision, an architecture bet. The surface transfers. The reliability under real conditions does not.

The uncomfortable asymmetry: distillation leaks the wrong things and drops the right ones

You might hope the losses are at least fair. That distillation transfers the good stuff and drops the incidental noise. The evidence points the other way, and this is the part that should make you uneasy.

A 2023 NeurIPS study with the blunt title "Students Parrot Their Teachers" showed that distilled students can still leak membership and memorization signals inherited from the teacher (Jagielski et al., 2023). Properties nobody was trying to transfer, specific memorized data, private signal, survived the process. Meanwhile the properties everyone wants, robust judgment and reliability, are the ones that tend to fall away.

So distillation is selective, but selective in the wrong direction. It can preserve what should have stayed private while dropping what should have been preserved. The accident survives; the skill leaks out.

Map that onto a person's published work and the picture gets sharper. What travels easily out of your writing is often the incidental specifics: the exact number you quoted, the phrasing of your recommendation, the particular example you used. What doesn't travel is the reason you chose that example over fifty others, or knew the number was load-bearing here but a rounding error there. The quotable surface propagates. The reasoning underneath stays put.

The human split runs along the same seam

Here is the extension that matters for anyone deciding what to publish. Human expertise, once it goes public and gets absorbed by AI systems, splits along the exact same seam as a distilled model.

What's reproducible transfers cleanly:

• A method you can lay out in steps.
• A checklist someone else could follow.
• A documented decision with its stated rationale.
• A worked example from start to finish.

All of that generalizes and transfers well, for one simple reason: it was already explicit enough to write down. Writing it down is the same act as making it copyable. The moment it fit in a paragraph, it was distillable.

What's not reproducible is the judgment call. Which of a hundred technically-correct options is the right one in this specific, messy, high-stakes situation. That was never fully captured in the writing to begin with, so it doesn't transfer just because the surrounding method did. You can publish the framework and still keep the thing that tells you when the framework doesn't apply.

This is the tacit slice, and it's worth being concrete about what it's made of:

• Pattern recognition built from having been burned before.
• The judgment to know when a rule doesn't apply to the case in front of you.
• The confidence to say "this one's fine to ship, that one isn't" without being able to fully justify it in words.

Notice what protects that slice. It isn't secrecy. You could try to explain it and you'd still fall short, because the knowledge lives below the level where language operates. It's protected by the fact that it was never writable to begin with. That is a far more durable moat than anything you keep behind a paywall, because there's nothing to leak.

You can't manufacture judgment from more copies

The obvious counter is scale. If judgment doesn't transfer cleanly, can't we generate enough synthetic examples of it, feed the model more, and close the gap statistically? Just make more data.

The evidence says no, and it says so at a systemic level. Nature published work in 2024 documenting what researchers call model collapse: AI models progressively degrade when trained recursively on data generated by earlier models rather than grounded in real, original sources (Shumailov et al., Nature, 2024). The degradation hits the tails first. The models lose coverage of rare and unusual cases, the exact edges where judgment earns its keep.

That's the mechanism in one line. Manufacturing more "judgment" synthetically, without grounding it in real, lived cases, degrades the signal rather than replacing it. The messy, expensive, hard-won cases are the ones that matter most and the ones synthetic generation reproduces worst. Real expertise isn't a substitutable input you can print more of. It's the grounding that keeps the whole system from drifting into a blurry average of itself.

The paradox is real, but bounded

So we can resolve Part 1's tension, partially. The expertise paradox is real. When you publish, the commons genuinely absorbs the articulable slice of what you know, and it will get remixed, generalized, and served back to the world at zero marginal cost. That's not paranoia. It's how distillation works, applied to prose.

But the paradox is bounded, and the boundary is precise. The commons absorbs the method. It does not absorb the judgment, because the judgment was never in the text. You are training the commodity, not the moat.

That reframing changes the strategic question entirely. If publishing gives away the commodity while leaving the moat intact, the defensive instinct from the top of this piece, hide the method, is optimizing for the wrong thing. You'd be protecting the part that copies anyway and neglecting to build the part that can't.

Which is exactly the question Part 3 takes up, in a post titled "Publish Anyway." If the moat is the tacit judgment, and if publishing the method is close to costless in strategic terms, how do you deliberately build that moat, and just as importantly, how do you signal it to the people who can't see it from the outside? That's the strategic response, and it's where this series is headed.

FAQ

What is the retention assumption in knowledge distillation?

The retention assumption is the unstated logical leap that if a smaller student model matches a larger teacher model's benchmark score, it must have retained the teacher's underlying capability. A 2026 position paper (Wang, arXiv:2604.25110) named this gap and argued it's false: students routinely match headline scores while diverging on calibration, robustness, privacy behavior, and reasoning faithfulness. Matching the output is not the same as inheriting the capability that produced it.

How big is the gap between a distilled model that "scores the same" and one that "behaves the same"?

Large, and mostly invisible to standard tests. A 2025 metamorphic-testing study of distilled code-generation models (Awal, Rochan and Roy, arXiv:2511.05476) found students preserved conventional accuracy while showing up to a 285% greater performance drop than their teacher under behavior-preserving transformations. Ordinary accuracy-based evaluation missed the behavioral discrepancies in up to 62% of cases. In a domain with an objective compiler, the gap was still wide and largely hidden.

Why doesn't publishing my method give away my real expertise?

Because your method and your judgment split apart the same way a distilled model's outputs split from its capability. The method transfers cleanly because it was already explicit enough to write down, which is the same thing as being copyable. The judgment, knowing which technically-correct option fits this specific situation, was never fully captured in the writing. It isn't protected by secrecy. It's protected by being unwritable in the first place.

Can AI systems just generate synthetic examples to learn judgment at scale?

Not reliably. Nature published evidence in 2024 (Shumailov et al.) of model collapse: models trained recursively on model-generated data instead of real, original sources progressively degrade, losing coverage of rare and unusual cases first. Those edge cases are exactly where judgment matters most. Manufacturing judgment synthetically, without grounding in real lived cases, degrades the signal rather than replacing it.

What's the practical takeaway for deciding what to publish?

Publishing gives away the commodity (the method) while leaving the moat (the judgment) largely intact, because the judgment was never in the text. The common defensive instinct of hiding the method protects the part that copies anyway. The more useful move is to build and signal the tacit judgment deliberately, which is the subject of Part 3, "Publish Anyway."

Further reading

• Wang, W., "Knowledge Distillation Must Account for What It Loses" (2026) — Position paper arguing that matching a teacher's benchmark score does not imply preserving the teacher's underlying capabilities; proposes a taxonomy of off-metric distillation losses. arxiv.org
• Awal, Rochan & Roy, "A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code" (2025) — Distilled code models showed up to 285% greater performance drop under adversarial transformation and up to 62% behavioral discrepancies invisible to accuracy-based evaluation. arxiv.org
• Jagielski et al., "Students Parrot Their Teachers: Membership Inference on Model Distillation" (NeurIPS, 2023) — Distilled student models can still leak membership/memorization signal inherited from their teacher. arxiv.org
• Shumailov et al., "AI Models Collapse When Trained on Recursively Generated Data," Nature (2024) — Recursive training on model-generated data progressively degrades models, with disproportionate loss of rare/tail-case coverage. nature.com