Domain Adaptation Versus Fine-Tuning: Adapting Is Not Retraining
Fine-tuning on new data and genuinely adapting a model to a new domain look similar but solve different problems, and confusing them quietly wrecks your evaluation.
Why the distinction gets blurred
In practice, teams often say 'we fine-tuned the model on the new domain' when what they actually did was retrain on a fresh dataset and call it a day. Fine-tuning is a training procedure: you take a pretrained model and continue updating its weights on new labelled examples. Domain adaptation is a problem statement: your model was trained on one distribution and must perform well on a related but different one, often with limited or no labels in the target domain. These are not the same thing, and treating them as interchangeable leads to evaluation setups that quietly answer the wrong question.
The confusion matters because fine-tuning implicitly assumes you have enough labelled target-domain data to make ordinary supervised learning work. If you have that, you barely need the vocabulary of domain adaptation at all: you just have a slightly awkward extra training stage. Domain adaptation earns its name precisely in the regime where labelled target data is scarce or absent, and the interesting question becomes how much you can transfer from the source domain without collecting fresh labels. Collapsing the two hides this assumption, and hidden assumptions are where reproducibility problems live.
I have seen this play out as follows: a model is trained on formal, edited text, then 'adapted' by fine-tuning on a modest sample of informal, user-generated text drawn from the same period and platform as the test set. The reported improvement looks like evidence that fine-tuning solved the domain shift. In reality, the fine-tuning set and the test set share vocabulary, topics, and even specific phrasings that never appear in the true deployment distribution. The model has not learned to generalise across domains; it has memorised a narrow slice of one.
A worked example with numbers
Suppose a sentiment classifier is trained on product reviews and achieves 91 percent accuracy on a held-out set of product reviews. We want it to work on restaurant reviews instead. Someone fine-tunes the pretrained product-review model on 2,000 labelled restaurant reviews, then reports 89 percent accuracy on a restaurant-review test set. This looks like a strong result: barely any drop moving domains, and only a small fine-tuning set was needed.
Now look at how the 2,000 restaurant reviews and the test set were collected. If both were scraped from the same handful of restaurant chains in the same six-month window, the model may have learned chain-specific phrasing, recurring menu items, and even repeated reviewer usernames rather than anything general about restaurant sentiment. Swap the test set for reviews from a different city, a different cuisine mix, or a later time period, and accuracy might fall to 76 percent. That 13-point gap is the real domain adaptation problem; the 89 percent figure only measured how well fine-tuning fits a narrow, correlated sample.
Contrast this with a genuine domain adaptation setup: no labelled restaurant reviews at all, only unlabelled restaurant text alongside the labelled product reviews. Techniques here try to align the two domains at the representation level, for instance by encouraging the model to produce features that a discriminator cannot use to tell product reviews from restaurant reviews, while still using the product-review labels to drive the classification task. If this approach reaches 81 percent on the genuinely held-out restaurant test set, that is arguably more impressive and more honest than the fine-tuned 89 percent, because it was achieved without peeking at labelled target data and without the risk of the target sample being unrepresentative.
The lesson is not that fine-tuning is bad; when labelled target data is plentiful and representative, it is usually the simplest and best option. The lesson is that a good fine-tuning number tells you almost nothing about domain robustness unless you have checked that the fine-tuning data and test data do not share the kind of narrow overlap that inflates performance.
What this means for evaluation design
If you are evaluating a claim of domain adaptation, the first question to ask is whether the target-domain data used anywhere in training, including fine-tuning, could share sources, time windows, authors, or templates with the test data. This is the same leakage concern that shows up in ordinary train-test splits, just dressed in domain-shift clothing. A split by domain label is not enough if the underlying collection process still ties training and test examples together.
Second, always report a source-only baseline: the model trained purely on the original domain, evaluated directly on the target-domain test set with no adaptation at all. This number anchors everything else. If fine-tuning only improves on it by a couple of points, the domain gap may not have been as large as assumed, or the fine-tuning set may be too similar to the test set to tell you much. If the improvement is large, you want to know whether it came from genuine transfer or from memorising overlap.
Third, when data allows it, test across multiple target sub-domains rather than one. A model adapted to restaurant reviews from one city should ideally be tested on restaurant reviews from several unrelated cities or time periods. Consistent performance across these slices is far more convincing than a single strong number, and it is cheap to arrange if you have planned your data collection with this in mind from the start rather than after the fact.
The practical takeaway
Fine-tuning and domain adaptation solve different problems dressed in similar clothes: one assumes labelled target data and asks how best to use it, the other assumes you barely have any and asks how much can be salvaged from the source domain. Before trusting a reported improvement from 'adapting' a model, check what data made that improvement possible and whether it could plausibly leak into the test distribution. A smaller, honestly measured gain from a properly held-out target set is worth more than a large one that turns out to be an artefact of a narrow, correlated fine-tuning sample. Retraining on new data is easy to do and easy to misread; genuine adaptation is the harder, more interesting claim, and it deserves an evaluation design that actually tests it.
