Why In-Domain Scores Overstate NLP Model Quality
A model that scores 94% on its test set can fall apart on text that looks only slightly different. Here is why that gap exists and how to measure it honestly.
The comfortable lie of a held-out test set
Most NLP papers and most internal reports lean on a simple ritual: split the data, train on one part, report accuracy or F1 on the other. This feels rigorous because it avoids the most obvious sin, training on your test set. But it quietly assumes something far stronger, that the test set is a fair sample of the conditions the model will meet in the world. That assumption is usually false, and the gap between what a held-out split measures and what deployment demands is where a lot of quiet failure lives.
Consider a sentiment classifier trained and tested on product reviews scraped from a single retail site. It might reach 93% accuracy on its held-out split. That number is real, but it only tells you how well the model captures the vocabulary, sentence length, and labelling conventions of that one site. Move the same model to customer support transcripts, or to reviews from a different market with different slang, and accuracy can drop into the 70s without any change to the model itself. The test score did not lie, but it answered a narrower question than most people think it answered.
The reason this keeps happening is that in-domain and out-of-distribution performance are genuinely different quantities, not the same quantity measured with more or less noise. In-domain accuracy tells you how well a model has fit the statistical regularities of one corpus: its topic distribution, its author style, its label noise pattern. Out-of-distribution performance tells you whether the model learned something that generalises beyond those regularities. Conflating the two leads teams to ship models that look production-ready on a dashboard and then degrade the moment real users with real variation start sending input.
A worked example of how the gap appears
Suppose you are building a named entity recognition model for news articles, trained on a few years of financial news. On a held-out split from the same source, it tags company names, people, and locations at 88% F1. That is a strong number by most standards. Now you point it at sports journalism or at regional news from outlets outside your training window. Entity boundaries that were unambiguous in financial prose, like multi-word company names next to ticker symbols, do not exist in the new domain, and the model has never had to learn general rules for segmenting unfamiliar proper nouns. F1 might fall to 65% or lower, not because the task changed but because the training distribution never taught the model to handle the new surface patterns.
What makes this tricky is that the in-domain number and the out-of-distribution number are both correct measurements, they are just measuring different things. The 88% score is an honest estimate of performance on text statistically similar to training data. It becomes misleading only when it is used, implicitly or explicitly, as a proxy for general competence. The mistake is rarely in the arithmetic of computing the score, it is in the inference drawn from it.
This also explains why benchmark leaderboards can be poor guides to real-world readiness. A model fine-tuned heavily on a benchmark's training distribution can edge out a more robust model on that benchmark's test set while being considerably worse on text drawn from a slightly different population. The leaderboard rewards fitting the benchmark's idiosyncrasies, not necessarily the underlying linguistic competence the benchmark was meant to proxy for.
Building out-of-distribution checks into evaluation
The practical fix is not to distrust in-domain scores entirely, they remain useful for comparing models under matched conditions, but to treat them as one data point among several. A more honest evaluation protocol deliberately constructs or sources test sets that differ from the training distribution along dimensions you expect to matter in deployment: different time period, different register, different source, different demographic of writer. If you can only build one extra evaluation set, choose the axis of variation most likely to occur in production, since that is where the stakes are highest.
It also helps to be explicit about what kind of shift you are testing for, because out-of-distribution is not one thing. Shift in topic is different from shift in writing style, which is different again from shift in label distribution. A model can be robust to one and fragile to another. Reporting a single aggregate robustness number without naming the shift tested hides exactly the information a reader needs to judge whether the result transfers to their own use case.
Leakage-aware splitting matters here too. If your out-of-distribution set accidentally shares authors, templates, or near-duplicate passages with the training set, you have not tested generalisation at all, you have tested memorisation under a different label. I have seen supposedly out-of-distribution evaluation sets that were scraped from the same forum threads as the training data a few months apart, which produces a reassuring score that tells you almost nothing about genuine robustness.
Finally, treat the gap itself as a measurement worth reporting, not just the raw out-of-distribution score. A model that drops from 90% to 85% across a domain shift is behaving very differently from one that drops from 90% to 60%, even if their in-domain numbers were identical. That delta is often more informative for deployment decisions than either number alone, because it tells you how much performance you are likely to lose when the real world inevitably differs from your training corpus.
The practical takeaway
In-domain accuracy is a valid but narrow measurement, and the discipline of good evaluation is knowing exactly how narrow. Before trusting a headline score, ask what distribution it was computed on and whether that distribution resembles the one your model will actually face. Where possible, build at least one deliberately mismatched test set, document the kind of shift it represents, and report the performance gap alongside the raw numbers. A model that is honest about its weaknesses on unfamiliar text is far more useful than one that looks perfect until it meets the real world.
