Comparing Two ML Models Without Fooling Yourself
One accuracy number beating another by a point means nothing on its own. Here is how to check whether a difference between models is real or just noise from your test split.
Why a single number is not enough
I have lost count of the times I have seen a table in a paper or a report where Model B beats Model A by half a percentage point on accuracy, and the conclusion is written as if that settles the matter. It rarely does. Test sets are finite samples. If your test set has a thousand examples, a difference of five correct predictions can easily be the kind of wobble you would get just from reshuffling which examples happened to land in the split.
This matters because model comparison decisions get baked into real choices: which architecture to ship, which paper claim to trust, which baseline to beat in a thesis. If the difference you are celebrating is indistinguishable from noise, you are not making a decision based on evidence, you are making it based on the specific random draw of your test set. Statistical significance testing gives you a disciplined way to ask: is this difference big enough, given the amount of data I evaluated on, that I should believe it reflects something real about the models rather than sampling luck?
The honest answer is often uncomfortable. Many published improvements in machine learning are small enough that a proper significance test would call them inconclusive on the reported test set size. That is not a reason to abandon testing, it is a reason to test properly rather than skip the question entirely.
Picking a test that matches your evaluation, not a generic one
The mistake I see most often is using a test designed for comparing two independent samples, such as a plain t-test on two separate accuracy figures, when the two models were evaluated on the exact same test examples. That independence assumption is wrong here, and it wastes information. If both models see the same items, you want a paired test that looks at where they disagree, not a test that compares two summary statistics as if they came from unrelated populations.
For classification with a single test set, McNemar's test is the classic tool. It looks only at the examples where the two models disagree: how many did Model A get right and Model B get wrong, and vice versa. Suppose out of a thousand test examples, both models agree on 900 of them, entirely irrelevant to the comparison. Of the remaining 100, Model A alone is correct on 65 and Model B alone is correct on 35. McNemar's test asks whether a 65 to 35 split is surprising under the assumption that disagreements should go either way with equal probability. That split is quite lopsided, and it would typically produce a p-value comfortably below 0.05, giving you reasonable grounds to say Model A's edge is unlikely to be pure chance. Compare that with a 55 to 45 split on the same 100 disagreements: far less convincing, and the test would usually tell you so.
If you have run repeated cross-validation or multiple training seeds, a paired t-test or a Wilcoxon signed-rank test across the per-fold or per-seed score differences is more appropriate than McNemar's, because now your unit of comparison is the fold-level score rather than the individual prediction. For regression or continuous metrics, the paired bootstrap is often the most flexible option: resample the test set with replacement many times, compute the metric difference each time, and see how often that difference crosses zero. If, across ten thousand bootstrap resamples, the difference favours Model A in 9,600 of them, you have a strong signal; if it is closer to 5,300, you do not.
The traps that make significance tests lie to you
A significance test is only as trustworthy as the independence of your evaluation data, and this is where leakage quietly ruins everything. If your test examples are not truly independent draws, duplicate rows, near-duplicate images, multiple samples from the same patient or the same user, then your effective sample size is far smaller than the number of rows suggests, and the test will report a confidence it has not earned. I would rather run a test on 300 genuinely independent examples than on 3,000 examples with hidden clusters, because the second number is a comforting illusion.
Multiple comparisons are the second trap. If you test five model variants against a baseline and report the one significant result at the conventional 0.05 threshold, you have not found a real effect, you have found what you would expect from chance alone roughly one time in twenty when running five tests. Correcting for this, even with something as simple as a Bonferroni adjustment, is not optional once you are hunting across several candidate models or several metrics.
Finally, remember that statistical significance and practical significance are different questions. With a large enough test set, a genuinely trivial difference, say a tenth of a percentage point in accuracy, can become statistically significant. That does not mean it is worth the added complexity, latency, or training cost of the fancier model. Always ask whether the effect size, not just the p-value, is large enough to matter for your actual use case.
A practical checklist for honest comparisons
Before I trust a claim that one model beats another, I check a short list: were the models evaluated on the same held-out data, ideally the same exact examples; is the comparison test paired rather than treating the two models as independent samples; is the test set free of duplicates or leaked groups that inflate the apparent sample size; and has the number of comparisons been accounted for if several models were tried.
In practice, my workflow is to compute the paired disagreement counts for classification tasks and run McNemar's test as a quick sanity check, then fall back to a paired bootstrap over an appropriate metric when the outcome is continuous or when I have run multiple seeds. I report the p-value alongside the raw effect size and, where possible, a confidence interval on the difference, because a confidence interval tells a fuller story than a single threshold decision ever can.
None of this replaces good judgement. A test can only tell you whether an observed difference is likely to be noise given your assumptions; it cannot tell you whether your test set represents the real deployment distribution, or whether the better model on paper is worth the engineering cost in practice. Use significance testing as one honest filter among several, not as a rubber stamp, and be just as suspicious of your own promising results as you would be of someone else's.
