A Worked Example of Overfitting to the Leaderboard

Why this matters

Most people learn about overfitting as a training set problem: a model memorises noise instead of signal, and its performance on held-out data collapses. That lesson is correct but incomplete. There is a second, quieter form of overfitting that happens even when your model never sees the test labels directly. It happens when you, the researcher, see the test score too many times.

I mean this literally. Every time you submit a model to a leaderboard, tweak a hyperparameter, and resubmit, you are running a search procedure. The leaderboard score becomes the objective you are optimising, whether you intend that or not. Given enough iterations, you will find a configuration that fits the quirks of that particular test split rather than the underlying task. The model has not overfit the training set in the classical sense; the whole pipeline, including you, has overfit the evaluation set.

This matters because it is invisible from the inside. Classical overfitting shows up as a gap between training and validation accuracy, something you can plot and catch. Leaderboard overfitting shows up only later, when the model is deployed or evaluated on a fresh, truly unseen set, and the score quietly drops. By then the damage is done and the cause is hard to diagnose, because nothing in your training logs looks wrong.

A worked example

Suppose you have a held-out test set of 1,000 examples for a binary classification task, and a baseline model that genuinely performs at 70 percent accuracy. You do not know the true figure; you only see the leaderboard score after each submission, which carries some sampling noise. With 1,000 examples, the standard error on a proportion near 0.70 is roughly 1.4 percentage points, so a single evaluation can easily read anywhere from about 67 to 73 percent just from noise, even with no change to the model at all.

Now imagine you make twenty small, essentially arbitrary tweaks: a different random seed, a slightly different learning rate, a minor change to data augmentation, none of which meaningfully improves the true underlying accuracy. You submit all twenty and keep the best score. Because each submission carries about 1.4 points of noise, the maximum of twenty noisy draws from a distribution centred at 70 will typically land two to three points above the true mean, simply from extreme value statistics. You will report, with complete sincerity, that your best configuration reaches 73 percent, when the true expected performance of that configuration on fresh data is still close to 70.

This is not fraud and it does not require bad intent. It is the same logic as multiple comparisons in statistics: test enough hypotheses against the same fixed sample, and one of them will look significant by chance. A leaderboard is a fixed sample. Every submission is a comparison. The more comparisons you run against it, the more the best observed score diverges from the true score, and the gap grows roughly with the number of attempts, not with any real improvement in the model.

The practical sting comes when the model moves to a genuinely new test set, one that was not part of the selection process. There, the lucky two to three points of noise do not reappear, because that noise was specific to the original 1,000 examples and the particular configuration that happened to flatter them. The score reverts towards 70, and the drop looks like deployment failure, even though nothing about the model changed. It was never really at 73.

Why the mechanism is easy to miss

Part of what makes this trap persistent is that each individual decision looks reasonable. Trying a different learning rate is good practice. Checking the score after a change is good practice. It is only the accumulation, dozens of small reasonable decisions each informed by the same fixed evaluation set, that produces the bias. There is no single moment where you can point and say that was the mistake, which is exactly why teams repeat it across projects.

It also helps to notice that the problem scales with how small and how reused the evaluation set is. A test set of 1,000 examples is far more vulnerable to this effect than one of 100,000, because the sampling noise per evaluation shrinks as the set grows, so the maximum-of-many-attempts inflation shrinks with it. Public benchmarks that have been used by thousands of submissions over years are a particularly severe case: even if each individual team acts carefully, the collective search across the whole community can push reported state-of-the-art numbers upward partly through selection rather than genuine progress.

The same mechanism appears inside a single project under the guise of validation set tuning. If you use one validation split to choose among many model variants and then report that validation score as your final number, you have made the validation set function as a second training set. The fix people usually reach for, a separate test set checked once at the very end, only works if it is genuinely checked once. Checking it twice and keeping the better of the two results reintroduces the exact same bias on a smaller scale.

What to actually do about it

The honest discipline is to treat every look at a test or leaderboard score as spending a limited resource. Decide in advance how many submissions or evaluations you will allow, keep a running log of what changed between attempts, and resist the urge to cherry-pick the best run after the fact and present it as representative. If you must iterate many times, do your comparisons on a separate validation split and reserve the test set strictly for a final, single-shot confirmation.

When reporting results, it is far more useful to show the distribution of outcomes across seeds and configurations than the single best number you happened to achieve. A mean and spread across five or ten runs tells a reader something stable; a cherry-picked maximum tells them almost nothing except how many attempts you made. If a leaderboard or benchmark is central to your work, read its documentation for how it limits submissions, because many well-designed ones cap the number of times a team can query the test server for exactly this reason.

The underlying idea is simple enough to carry around: a test set only tells you the truth the first time you ask it. Every subsequent question, if you let the answer influence your choices, nudges that truth a little further from reality. Overfitting is not only something your model does to your data; it is something you can do to your evaluation, and the only defence is counting your questions.