Designing an Evaluation Harness Before You Know the Job

The problem with evaluating for a moving target

Most evaluation advice assumes you already know the task: fixed dataset, fixed metric, fixed split. In practice, a lot of applied machine learning work starts earlier than that. You are asked to build something useful, the exact deployment scenario is still being negotiated, and the data you have today is a proxy for data you will see later. If you wait until the job is fully specified before thinking about evaluation, you end up bolting on a test set at the last minute, and that test set is usually too close to the training data to tell you anything honest.

I think the right move is to treat the evaluation harness as infrastructure you build early, independent of the exact model architecture, and deliberately robust to the fact that the final task will shift under you. This is not about predicting the future perfectly. It is about designing measurement scaffolding that degrades gracefully when assumptions change, rather than silently lying to you.

A concrete example helps. Imagine you are asked to build a model that flags unusual transactions for a client, but the client has not yet decided whether it will be used for real time screening, daily batch review, or quarterly audit sampling. Each of those implies different latency constraints, different label availability, and different cost trade-offs between false positives and false negatives. If you evaluate only against one implicit assumption, say batch review with balanced classes, you may ship something that performs terribly the moment it is repurposed for real time screening with a 1 in 2000 positive rate.

Separate the invariants from the assumptions

The first practical step is to write down, explicitly, what you believe will not change versus what might. Invariants might include things like: the underlying population of entities being scored, the rough feature availability, and the fact that recall on the rare class matters more than raw accuracy. Assumptions might include: the exact threshold, the exact cost ratio between error types, and whether labels arrive within a day or within a month.

Once you have that split, build your harness around the invariants and make the assumptions configurable rather than baked in. Concretely, this means your evaluation code should compute a full precision-recall curve and calibration curve rather than a single accuracy number at one threshold. It means you store raw predicted probabilities, not just binary decisions, so that if the deployment context changes the required threshold, you can recompute performance without retraining or re-scoring anything.

Here is a worked intuition. Suppose your model produces a score between 0 and 1, and on a held-out set of 10,000 cases with 50 true positives, you get an area under the precision-recall curve of 0.38. If the eventual job turns out to need a working point at 90 percent recall, you can read straight off your stored curve that precision at that recall is, say, 12 percent, meaning roughly 7 false positives for every true positive caught. If the job instead needs 99 percent precision, you might find that recall collapses to 20 percent at that operating point. Neither of these numbers is good or bad in isolation; they only become meaningful once someone tells you the actual cost structure. The point of the harness is that you did not need to know that cost structure in advance to generate the numbers that will answer the question later.

Guard against leakage before you know the exact split you will need

Leakage is the single most common way an evaluation harness quietly lies, and it gets worse, not better, when the task is underspecified, because you are more likely to reach for whatever split is convenient rather than the one that matches deployment reality. If your data has any grouping structure, such as multiple transactions per customer, multiple images per patient, or multiple sentences per document, you must split at the group level, not the row level, even if you are not yet sure whether the final job will care about generalising to new customers or new documents. Splitting at the row level when the real deployment question is generalisation to new groups will give you an optimistic number that has nothing to do with reality.

Time is the other classic leakage vector. Even if you do not know whether the final system will run in real time or in batch, if there is any temporal ordering in your data, use a time-based split by default. Train on the earlier period, evaluate on the later period. This is a conservative default: it is strictly harder to overfit to than a random split, and it remains valid whether the eventual job is real time scoring or retrospective analysis. The cost of assuming a random split is fine, when it later turns out temporal drift matters, is that your reported numbers were never achievable in production in the first place.

A useful worked check: take your existing random split evaluation and rerun it as a time-based split with the same model and features. If accuracy or AUC drops by a small amount, say a few percentage points, that is a healthy sign of mild distribution shift over time. If it drops sharply, from an AUC of 0.91 to 0.68 for instance, that is a strong signal that your original number was inflated by leakage, and you have just saved yourself from reporting a fantasy figure to whoever eventually specifies the real job.

Build in comparison baselines that survive task changes

Finally, an evaluation harness for an unknown job needs baselines that remain meaningful regardless of what the task turns out to be. A strong constant-prediction baseline, a simple rule-based baseline, and a basic linear or tree model baseline should all be run through the exact same harness as your main model. These baselines cost little to compute and they anchor your numbers: an AUC of 0.75 sounds respectable until you learn that a simple threshold on one feature already achieves 0.72, at which point your fancy model is buying very little.

Keep these baselines versioned alongside the harness itself, not as a one-off script you ran once. When the job specification finally arrives and someone asks for performance under a new operating point or a new subgroup, you want to be able to rerun everything, including the baselines, in an afternoon rather than a week. That reproducibility is the actual deliverable. The model will change, the task will be refined, the client's mind will change twice more before launch, but a harness that stores raw scores, respects grouping and time structure, and always reports against honest baselines will keep telling you the truth no matter what the job turns out to be.