Reproducibility: Seeds, Configs, and Runs You Can Rerun

Why 'it worked on my machine' is not enough

Every so often I read a result that looks impressive, a model beating a baseline by a comfortable margin, and my first question is not whether the number is correct but whether anyone, including the author six months later, could get that same number again. In my experience this is the quiet failure mode of applied machine learning. Nobody falsifies results on purpose. They just lose track of which random seed was active, which version of a preprocessing script produced the training file, or which of three near-identical config files was actually used for the run that got reported.

Reproducibility sounds like a hygiene issue, something you tidy up after the real work is done. I think that framing is backwards. If you cannot rerun an experiment and get the same, or statistically the same, outcome, you have not actually finished the experiment. You have produced a single uncontrolled observation. Treating reproducibility as a first-class part of the method, not an afterthought, changes how you write code, how you log runs, and how much you trust your own conclusions.

The good news is that most of the failure is mundane and fixable. It is rarely a deep philosophical problem about the nature of randomness. It is usually three specific, boring gaps: seeds that are not actually fixed everywhere they need to be, configuration that lives in someone's head or a Jupyter cell instead of a file, and a run history that cannot tell you what code and data produced a given number. Fix those three and you have solved most of the practical problem.

Seeds: fixing the ones you forgot about

Setting a random seed feels like a solved problem, you call one function and move on. The trouble is that a typical training pipeline draws randomness from several independent sources, and setting one does not set the others. Consider a fairly ordinary image classification run: the data loader shuffles the training set, the model initialises its weights, dropout masks are sampled during training, and if you are using a GPU, certain operations are non-deterministic by default for speed. If you fix the seed for your deep learning framework but not for the underlying array library, or you fix both but leave the data loader's shuffling on a separate unseeded generator, you can rerun the 'same' experiment and land on a validation accuracy that differs by a percentage point or two for no reason you control.

That gap matters more than it sounds. Suppose two model variants differ in true performance by half a percentage point, a modest but real improvement. If your own unseeded randomness contributes a swing of one to two points between runs of the identical configuration, that noise is larger than the effect you are trying to measure. You will happily report whichever run landed favourably and call it an improvement, when in fact you have measured seed variance, not model quality. This is a specific, avoidable version of the more general problem where noise gets mistaken for signal.

The fix is unglamorous but complete: set the seed for every source of randomness your pipeline touches, including the language's own random module, the numerical library, the deep learning framework, and the data loading or shuffling mechanism, and where relevant force deterministic algorithm selection even at some cost to speed. Then go a step further and run the exact same configuration three to five times with different seeds deliberately, not to get a lucky number but to see the spread. Reporting a mean and a spread across seeds is far more honest than reporting a single best run, and it tells you immediately whether the effect you found is bigger than the noise floor of your own pipeline.

Configs: making every decision an artefact, not a memory

The second gap is configuration. It is common, especially under deadline pressure, to have the 'real' settings for a run scattered across command-line flags typed once and never saved, a few lines edited directly in a script, and a mental note about which checkpoint corresponds to which learning rate. Three weeks later nobody, including the person who ran it, can reconstruct exactly what produced the reported result.

A workable habit is to treat the full configuration for a run, learning rate, batch size, number of epochs, optimiser settings, data split identifiers, model architecture choices, and the seed itself, as a single structured file that gets saved alongside the run's outputs, not edited in place for the next experiment. Concretely, if you are comparing five learning rates, that is five saved configuration files and five corresponding output folders, not one file edited five times with the previous values lost. This feels like overhead when you are moving fast, but the alternative is that your only record of what worked is a plot with no caption explaining how it was produced.

This matters even more once data preprocessing enters the picture, because a config should also pin down which version of the data and which split you used. A model trained on 'the training set' is not reproducible if that phrase can refer to three slightly different files created at different points as you cleaned the data. Recording a hash or version tag for the dataset file, alongside the exact indices or filter used to build the split, closes a gap that seeds alone cannot fix: you can have a perfectly deterministic training run on the wrong or shifted data and still get a number nobody can reproduce, because nobody recorded which data actually went in.

Runs you can actually rerun

The third gap is the run log itself. A reproducible experiment needs a record connecting a specific code version, a specific config, a specific data version, and the resulting metrics, all in one place. Without that link, even perfect seeding and perfect configs are useless, because you cannot trace a reported number back to the exact combination that produced it. A simple, low-tech version of this is a spreadsheet or log file where every run gets a unique identifier, a timestamp, a note on the code commit used, the path to its saved config, and the final metrics. A more automated version uses an experiment tracking tool, but the underlying discipline is the same regardless of tooling: every number you are willing to put in a report or a chart should be traceable to an exact, rerunnable combination of code, config, data, and seed.

The payoff shows up exactly when you need it least: during a deadline, when a reviewer or colleague asks why a number changed, or six months later when you want to extend the work. If your pipeline is genuinely rerunnable, you answer that question by rerunning it, not by guessing. That single capability, being able to regenerate a result on demand rather than defending it from memory, is what separates a robust piece of empirical work from a plausible-looking one. It costs a little discipline up front. It is far cheaper than the alternative, which is not trusting your own numbers.