Fine-Tuning, Distillation, and Model Adaptation

The most expensive misconception in this field is that fine-tuning is how you teach a model your company data. It is not, and understanding why saves months.

Supervised fine-tuning adjusts weights so that, given inputs like the ones in your training set, the model produces outputs like the ones in your training set. What it learns most reliably is form: structure, tone, register, task framing, output schema, domain vocabulary, and the decision boundary of a narrow classification. Show it enough examples of turning a support ticket into a strictly formatted triage record and it will get good at that shape.

What it learns unreliably is fact. Train on question-and-answer pairs about your internal systems and the model learns the pattern of confidently answering questions about your internal systems. When a question arrives that is close to but not in the training data, it produces something in the right style and the wrong substance. You have not added knowledge; you have added fluency about a domain, which is worse than useless because fluency is exactly what stops a reader noticing the error. Facts that change belong in retrieval, where you can update them without a training run.

There is a second distinction the course leans on throughout: capability versus elicitation. If a base model has genuinely never learned to do something — a reasoning pattern absent from pretraining, a language it has barely seen — a modest fine-tuning set will not create the ability. What fine-tuning does well is reliably surface a behavior the model can already produce sometimes. Before spending on data collection, it is worth checking whether careful prompting produces the target behavior occasionally. If it never does, adaptation on a small dataset is unlikely to conjure it. If it does so occasionally but not dependably, you have something worth making consistent.

The cost that is easy to forget

Adaptation also removes something. Training narrowly on one task degrades performance elsewhere, a phenomenon usually called catastrophic forgetting. A model tuned hard on JSON extraction becomes a worse conversationalist. Usually that is an acceptable trade, but only if you noticed it happened, which requires evaluating general ability and not only the target metric. Research published by groups at Princeton University and Virginia Tech has also shown that fine-tuning an aligned model can weaken its safety behavior even when the training data is entirely benign — a side effect worth measuring deliberately rather than assuming away.

Adaptation techniques form a rough ladder ordered by cost and irreversibility. Working down it in order is boring advice and it is correct.

Prompting and few-shot examples. Free to change, instantly reversible, and much stronger than people assume once you invest in structured instructions and well-chosen exemplars. Most teams give up here far too early.

Constrained and structured decoding. If the problem is that output format is inconsistent, grammar-constrained sampling or schema-enforced decoding solves it directly. Fine-tuning for format when a constraint layer would do is a common and expensive detour.

Retrieval. If the problem is that the model does not know something, this is the answer. It is also the only approach where updating knowledge costs nothing.

Routing and cascades. Split the traffic rather than changing the model: let a cheap model attempt every input and escalate only the cases it flags as uncertain or that fail a validator. Where the workload is mostly routine, this captures a large share of the saving people expect from adaptation, and it can be switched off the day it stops helping.

Distillation. Use a strong model to generate training data, then train a small model on it. The right move when you have a fixed, narrow, high-volume task and the strong model already solves it.

Supervised fine-tuning on human-authored data. Now you are paying for annotation, which is where the real budget goes.

Preference optimization and grader loops. The most powerful and the most failure-prone. Reserve for behavior that cannot be specified as a single correct output.

When descending is justified

Four situations reliably justify the move to trained weights. Unit economics at volume: the fixed cost of data and training amortizes over enough calls that a smaller model wins. Latency: a small tuned model with a short prompt responds faster than a large one carrying a long system prompt. Deployment constraints: the workload must run on your hardware or inside a network boundary, and only a small model fits. And behavioral consistency: an output shape that prompting produces most of the time but not reliably enough for an automated downstream consumer.

Situations that do not justify it: knowledge that changes, a task you have not yet defined an evaluation for, low call volume, and the belief that a custom model is inherently more impressive. The last one funds a surprising number of projects.

One economic detail changes the calculation and is often missed. Adapter-based tuning does not require a dedicated deployment per task. Serving stacks that support multiple low-rank adapters over one shared base let many tuned variants share the same weights and the same GPU, with the adapter selected per request. That collapses the serving overhead that historically made small-scale fine-tuning uneconomic.

Between the dataset and the hyperparameters, the dataset decides the outcome, and it is not close. The course allocates its weight accordingly.

Quality dominates quantity past a surprisingly low threshold. The LIMA work published by Meta AI argued that a small, carefully curated and consistently styled instruction set can outperform a much larger noisy one. A modest set you have personally read through will usually beat a far larger scraped one. This is good news, because it puts adaptation within reach of a small team, and bad news, because there is no way to buy your way past the checking.

The failures that show up in the trained model

Every property of your dataset becomes a property of the model, including the ones you did not intend.

• Style leakage. If most of your answers open with the same phrase, the model will open with it always, including where it makes no sense.
• Length imitation. Models copy the length distribution of their targets. Train on verbose answers and you get verbosity you then pay for on every inference call.
• Distribution mismatch. Training inputs are usually cleaner than production inputs. If real users send typos, truncated pastes and mixed languages, and your training set contains none of that, the model degrades exactly where it matters.
• Missing negatives. A dataset containing only successful task completions teaches a model that every input is completable. It will never refuse, never ask for clarification, and never say the document does not contain the answer. Those behaviors have to be in the data.
• Template mismatch. The single most common silent bug in the whole discipline. Train with one chat template and serve with another and the model appears mysteriously worse. Verify the exact serialized string on both sides, byte for byte.
• Contamination. Any overlap between training and evaluation sets makes your metrics fiction. Deduplicate across splits with something stronger than exact string match — near-duplicate detection over normalized text.

Annotation is specification

Writing the labeling guideline forces you to decide what correct output actually is, and that decision is the real product spec. Two annotators disagreeing is not a nuisance, it is a signal that the task is underspecified — and a model trained on inconsistently labeled data learns to be inconsistent. Measuring agreement before scaling up annotation is cheaper than discovering the ambiguity after training.

Synthetic data changes the economics but not the principles. A strong model can generate candidate examples far faster than humans can write them, and rejection sampling — keeping only generations that pass a verifier, a test, or a schema check — turns raw generation into usable supervision. What it cannot do is invent the judgment about what good looks like. It also carries a constraint people skip: the terms of service of hosted models frequently restrict using their outputs to train competing systems, and open-weight models carry licenses with their own conditions. Read them before building a pipeline on top of one.

Once you have decided to train, the method question is mostly about memory and reversibility.

Full fine-tuning updates every parameter. Memory is dominated not by the weights but by the optimizer state and gradients that accompany them, which is why the practical requirement is several times the size of the model itself. It gives the largest capacity to change behavior, and correspondingly the largest capacity to break things. It also produces a complete new model per task, which multiplies storage and serving cost.

LoRA, published by Microsoft Research, freezes the base weights and trains small low-rank matrices injected alongside selected projections. The trainable parameter count drops by orders of magnitude, the checkpoint is small enough to move around casually, and the base model is untouched — so the change is reversible by not loading the adapter. The main knobs are the rank, the scaling factor, and which modules you target. Too low a rank underfits tasks that require real behavioral change; very high ranks approach full fine-tuning in cost while losing the compositional benefits. Targeting attention projections alone is the common default, though including the feed-forward projections often helps on harder adaptations.

QLoRA, published by researchers at the University of Washington, adds quantization of the frozen base to four bits, with adapters trained in higher precision on top. This is what put fine-tuning of large models on single consumer GPUs. The trade-off is real and worth stating: the base you trained against was quantized, so merging the adapter back into a full-precision base does not reproduce the same model. Serve the configuration you trained, or re-evaluate after any change to it.

The knobs that actually matter

The course is opinionated about where attention pays off. Learning rate matters most and behaves differently for adapters than for full tuning — adapters tolerate and generally need higher rates. Epoch count is where small datasets die: overfitting arrives quickly, and the symptom is a model that reproduces training answers verbatim while degrading on anything slightly different. Loss masking is easy to get wrong and consequential — computing loss over the prompt as well as the completion teaches the model to generate prompts, which is rarely the goal. Sequence packing improves throughput but corrupts training if attention is allowed to cross document boundaries.

Everything else is secondary until these are right. The instinct to sweep a dozen hyperparameters before fixing the data is the most reliable way to spend a large compute budget learning nothing.

Distillation is the technique with the clearest business case and the most under-discussed pitfalls.

The original formulation, introduced by Hinton, Vinyals and Dean at Google, trained a small student to match the full output distribution of a large teacher rather than only its top answer, on the reasoning that the relative probabilities of wrong answers carry useful information. That approach remains valid when you control the teacher and can read its logits.

Most LLM distillation in practice looks different, because the strong teacher is usually behind an API that returns text and nothing else. The working pattern is response distillation: run the teacher across a large set of realistic inputs, keep the good outputs, and supervise the student on the resulting pairs. Add the intermediate reasoning of the teacher to the targets and you get rationale distillation, which tends to transfer multi-step behavior better than answers alone — although it also inflates output length, and you may want the student to reason less verbosely at inference than it was trained to.

Filtering is the whole trick

Naive distillation copies the errors of the teacher along with its skills, and the resulting student is worse than an obviously weak model because it fails fluently. Confident mistakes made by the teacher become confident mistakes made by the student, expressed in the same authoritative register.

Where a verifier exists, use it. Code that must pass tests, arithmetic that must check out, extraction that must validate against a schema, SQL that must execute — in all of these you can generate many candidates and keep only the ones that pass. This rejection-sampling loop turns a mediocre teacher into a good dataset, and it is the reason distillation works far better on verifiable tasks than on open-ended ones.

Where no verifier exists, you need either human review of a sample large enough to characterize the error rate, or a second model acting as a critic, or both. Distilling an open-ended task with no filtering step is where the disappointing results come from.

Choosing what to distill into

Student capacity has to be matched to the task. A narrow, well-defined transformation distills into a small model comfortably. Broad general capability does not — you cannot compress the general reasoning of a frontier model into a tiny one and expect it to hold outside the distribution you trained on. The honest framing is that distillation buys you a specialist, and the value comes from the specialist being enough for the job. When teams are disappointed by distillation, the usual cause is that they were quietly hoping for a generalist.

Supervised fine-tuning needs a single correct output per input. Plenty of behavior cannot be expressed that way. Which of two explanations is clearer, which refusal is appropriately calibrated, which summary is more useful — these are comparative judgments, and preference methods exist to train on comparisons rather than on targets.

The original production pipeline, popularized by the OpenAI work on instruction-following models, trains a separate reward model on human comparisons and then optimizes the policy against it with reinforcement learning. It works and it is operationally heavy: two models, an unstable optimization, and a reward model that can be exploited by the policy it is meant to guide.

Direct preference optimization, published by researchers at Stanford University, showed that the same objective can be reached by optimizing the policy directly on preference pairs, with no separate reward model and no reinforcement learning loop. It is far simpler to run, which is why it became the default entry point. Variants relax its data requirements in useful ways — some need only a binary good-or-bad label per sample rather than matched pairs, which is a much easier thing to collect from production feedback.

Where preference data comes from, and what it does to you

Human comparison data is the best and the most expensive, and it requires the same specification discipline as any other annotation. Model-generated feedback is cheaper; Anthropic published an approach using written principles and model critiques to produce preference data at scale, and variations of it are now common. Production signals — thumbs, edits, retries, abandonment — are free and badly biased toward whoever bothers to click.

Two failure modes appear reliably enough to plan for. The first is reward hacking: the policy finds a way to score well that has nothing to do with the intent. Given a judge that likes thorough answers, models get longer. Given a judge that likes confident answers, models get less calibrated. The verbosity drift after preference tuning is so consistent that length should be tracked as a metric in its own right.

The second is drift away from the competence of the base model. Preference optimization pulls hard, and unconstrained it will degrade general capability while improving the preferred behavior. The standard defense is a penalty term that keeps the tuned policy close to the reference model, and setting that constraint too loose is the most common way a preference run produces a model that wins comparisons and fails everywhere else.

Where a programmatic grader exists — tests, checkers, validators — reinforcement-style loops against that grader become viable and often outperform preference data, because the signal is objective. The caveat is identical to the one in every optimization problem: the model optimizes the grader, not your intent. A gameable grader will be gamed, and the more capable the model the faster it finds the gap.

Build the evaluation before the dataset. This ordering is not a preference; without it there is no way to know whether a training run helped, and the failure is silent because a fine-tuned model always looks different and different reads as better.

A workable evaluation setup has three layers. A held-out set from the same distribution as training, checked for contamination, which tells you whether learning happened. A set of realistic production inputs, including the ugly ones, which tells you whether it generalizes. And a general-capability regression suite unrelated to the task, which tells you what you broke. Teams routinely build the first, sometimes build the second, and almost never build the third, which is why forgetting is usually discovered by a user.

Comparisons should be pairwise against the pre-adaptation baseline. Absolute quality scores from a model judge are noisy; "is this better than what we already had" is the question you actually need answered, and it is a much more stable one to ask. Two traps are specific to adaptation work. If the judge comes from the same family as the teacher you distilled from, it will tend to reward the student for sounding like its teacher, which is not the same as being right — pick a judge from a different lineage, or fall back to human labels for the cases that decide the release. And because tuned models drift longer, a judge that likes thorough answers will score the tuned model up for a change that is purely stylistic, so log output length beside every win rate and re-run a sample with the two candidates swapped before believing the verdict.

Ablate one variable at a time. The temptation, after a run that improved things, is to change the data mixture, the rank and the learning rate together for the next one. When that run is worse you have learned nothing about which change caused it. Adaptation work is slow largely because this discipline is unglamorous.

After deployment

An adapter is bound to the exact base checkpoint it was trained against. Version the base, the adapter, the tokenizer and the serving template as one unit, because changing any of them independently produces a model that loads without error and behaves differently. When a new base model ships, your adaptation does not transfer — budget for retraining as a recurring cost rather than a one-off.

Watch the input distribution rather than only the outputs. Degradation usually starts upstream: a new customer segment, a changed frontend that alters how text arrives, a new document format. The model has not changed; the world has. Keeping the pre-adaptation baseline available as a fallback route makes rollback a configuration change instead of an incident.

Finally, keep asking the uncomfortable question. Would a better prompt on a stronger base model, available now and not when you started, beat what you trained? Base capability improves faster than most fine-tuned specialists do, and the discipline to retire a model you invested in is part of the job.