Picking a Model Evaluation Technique for Your Actual Problem

Why the default is usually wrong

Most people learn one evaluation recipe early on: shuffle the data, split it eighty-twenty, run k-fold cross validation, report accuracy or an F1 score, and move on. It becomes a reflex rather than a decision. The trouble is that this recipe encodes an assumption, namely that every row in your dataset is independent and identically distributed, and that the deployment setting looks like a random sample from the same distribution as training. That assumption is often false, and when it is false the evaluation number you report is not measuring what you think it is measuring.

I have come to treat the choice of evaluation technique as a modelling decision in its own right, not an afterthought bolted on after the real work is done. The question to ask first is not 'which metric looks rigorous' but 'what does a new, unseen case actually look like in this problem, and how did my existing data get generated'. Answering that honestly usually points to one evaluation scheme and rules out several others, often including the one you were about to use by default.

This matters because a mismatched evaluation scheme does not just give you a slightly noisy number, it gives you a systematically biased one. A model can look excellent under the wrong scheme and then fail quietly once deployed, and by the time anyone notices, the cost has already been paid in wasted effort, bad decisions, or worse.

Worked intuition: the same dataset, three honest answers

Suppose you are building a model to predict whether a patient will be readmitted to hospital within thirty days, using records collected over three years from one hospital. There are several reasonable questions you could be trying to answer, and each demands a different split.

If your real question is 'how well does this model generalise to a new patient admitted next month', a simple random k-fold split is dangerous. Patients with chronic conditions appear multiple times in the dataset across different admissions, so a random split can place two visits from the same patient in both the training and test folds. The model then partly learns to recognise the patient rather than the underlying risk pattern, and your cross-validated accuracy, say ninety-one percent, is inflated. The honest fix is a grouped split, where every record belonging to a given patient id sits entirely in one fold. Rerun the same model under grouped k-fold and it is entirely plausible the accuracy drops to the low eighties. That drop is not a worse model, it is a more truthful number.

If instead your real question is 'how well will this model perform next year, trained only on past data', the grouping fix is not enough, because it still lets the model train on April data and test on January data from the same year, which is not how deployment works. Here you need a time-based split: train on year one and two, test on year three, and ideally walk the split forward in a rolling fashion to see whether performance is stable or decaying as conditions drift. A model that scores eighty-five percent under grouped k-fold but seventy-six percent under a forward time split is telling you something important, namely that the relationship between features and outcome is shifting over time, perhaps due to a change in hospital policy or recording practice.

If your real question is instead about a specific subgroup, say patients over seventy with diabetes, then neither of the above splits is sufficient on its own. You need stratified evaluation on that subgroup specifically, because an aggregate metric across the whole dataset can hide the fact that the model is mediocre precisely where it needs to be good. A model with ninety percent overall accuracy might sit at seventy percent on the subgroup that actually drives most of the readmission cost.

Matching the metric to the decision, not just the split

Getting the split right solves the leakage problem, but the metric itself is a second, independent decision, and it should be driven by what the number is used for downstream. Accuracy is a poor default whenever classes are imbalanced or the cost of different error types is unequal, which is most real problems. If readmission happens in eight percent of cases, a model that predicts 'no readmission' for everyone scores ninety-two percent accuracy while being useless. Precision, recall, and the trade-off between them, or a threshold-independent measure like the area under the precision-recall curve, generally tell you far more than accuracy in this setting, and precision-recall curves in particular are more informative than ROC curves when the positive class is rare.

The same logic applies in regression. Mean absolute error and root mean squared error answer different questions: RMSE penalises large errors disproportionately, which is right if occasional big misses are genuinely costlier, and wrong if they are not. If you are forecasting demand and a twenty-unit error on a slow day matters as much per unit as a twenty-unit error on a busy day, MAE is the honest choice; if large misses cause stockouts that cascade, RMSE's extra weighting is doing useful work rather than distorting the picture.

Finally, consider whether a single number is even the right output. For a time series model, a single held-out error figure hides whether errors grow over the forecast horizon, and for a classifier used to triage cases, a single F1 score hides where on the precision-recall curve you actually need to operate. Reporting a curve, a per-segment breakdown, or performance across a rolling set of time windows is more work than reporting one clean figure, but it is the difference between an evaluation that survives contact with deployment and one that does not.

A practical takeaway

Before you choose a split or a metric, write down two sentences: what does a genuinely new case look like relative to my training data, and what decision will this evaluation number actually inform. If new cases can share structure with old ones, such as the same patient, same user, or same store, group your splits around that structure. If new cases arrive after old ones in time, split by time and check for drift rather than trusting a single random split. If the classes or outcomes are imbalanced, or errors have asymmetric costs, pick a metric that reflects those costs rather than reaching for accuracy by habit. None of this is exotic; it is simply taking the time to match the evaluation to the problem you actually have, rather than the problem that a default scikit-learn call happens to assume.