Building Your Own Evaluation Harness Before a Framework

Why the shortcut costs you later

It is tempting, the moment you have a trained model, to reach for an evaluation framework. There are good ones: they give you leaderboards, pretty tables, standard metrics computed in a single call. The problem is that they also give you distance. You stop asking how the split was made, whether the metric aggregates fairly across classes, or whether your test set has quietly leaked into training through a shared identifier. The framework answers questions you never explicitly asked, and that is exactly the danger.

I have come to treat a hand-built evaluation harness as a forcing function. Writing the splitting logic, the metric computation, and the reporting by hand means every assumption has to be typed out. You cannot hide behind a default argument called shuffle=True without noticing that shuffling a time series before splitting is precisely how you manufacture an optimistic result. The friction is the point.

None of this is an argument against frameworks. Once you understand your problem, a mature library will save you time and reduce bugs. The order matters, though: understand first, automate second. Reversing that order is how teams end up reporting a headline accuracy number that nobody can explain six months later, because the pipeline that produced it was assembled from defaults nobody read.

A worked example: the leak you cannot see in a summary table

Suppose you are building a model to predict whether a customer will cancel a subscription within thirty days, using ten thousand historical rows collected over eighteen months. A framework, given a dataframe and a target column, will happily do a random eighty-twenty split and report an AUC of, say, 0.91. That number looks respectable. It is also very likely a lie.

Customers appear multiple times in this kind of dataset: once per billing cycle, once per support ticket, once per login event. A random row-level split scatters the same customer across both train and test. The model does not need to learn general cancellation behaviour; it can partially memorise customer-specific quirks from rows it saw in training and recognise them again in test. That inflates the score without telling you anything about how the model will behave on a genuinely new customer next quarter.

When you build the harness yourself, you are forced to decide the unit of splitting before you can write a single line. Do you split by customer ID, ensuring no customer appears in both sets? Do you split by time, training on the first fourteen months and testing on the last four, so the evaluation mimics the real deployment scenario of predicting the future from the past? In my experience, once you switch from a random row split to a customer-level, time-respecting split on a dataset like this, the AUC of 0.91 tends to fall, sometimes sharply, and that drop is not a bug. It is the harness telling you the truth for the first time.

A framework will implement whichever split you ask for, but it will not ask you the question. Building the split logic yourself means you cannot proceed without confronting it, and that is worth an afternoon of extra work.

What a minimal harness actually needs

A useful evaluation harness is smaller than people expect. At minimum it needs four pieces: a documented splitting rule, a fixed random seed with the seed value recorded alongside results, a small set of metrics chosen for the specific failure modes you care about, and a report format that always shows the metric alongside the sample size and class balance it was computed on. That last part matters more than it sounds. An F1 score of 0.82 on a test set with four hundred positive examples means something quite different from the same score on a test set with twelve.

Class imbalance is another place where hand-rolling pays off. If your positive class is five per cent of the data, accuracy will happily reward a model that predicts the negative class every time with a score around 0.95, which looks excellent and is worthless. Writing your own metric code means you decide, explicitly, to report precision, recall, and a calibration check instead, or a metric weighted by class frequency, rather than accepting accuracy because it was the first thing printed by a library's summary method.

The other habit worth building is treating the test set as something you touch exactly once per genuine model candidate, not once per hyperparameter tweak. A common failure is using the test set repeatedly to choose between five model variants, then reporting the best test score as if it were an honest estimate. It is not; it is the maximum of five noisy estimates, and it will overstate performance on the next batch of real data. A harness that logs every evaluation call, with a timestamp and the model version, makes this pattern visible. You will see the same test set being queried nine times in an afternoon and realise you have quietly turned it into a validation set without meaning to.

The practical takeaway

Building your own harness is not about rejecting tools. It is about earning the right to use them. Once you have written the split logic, the metric code, and the logging by hand for a couple of projects, you develop an instinct for where frameworks hide their assumptions, and you know exactly which defaults to override the moment you adopt one.

The concrete habit I would recommend: before your next project touches a framework, write a plain function that takes your raw data, splits it by the correct unit (customer, time, session, whatever prevents leakage), computes two or three metrics chosen for the actual decision the model will drive, and prints the sample size next to every number. If that function embarrasses your first model's score, it has already done its job.