Evaluation Metrics: Interview Talk vs What You Actually Compute

Why the glossary answer is never the whole answer

Ask most candidates to define precision and recall and you get a clean, correct sentence. Precision is the fraction of predicted positives that are actually positive. Recall is the fraction of actual positives that the model caught. These definitions are true, memorable, and almost useless on their own, because the number you get out of them depends entirely on decisions nobody asks about in the thirty seconds you have to answer.

I think this gap matters because it is where junior and experienced practitioners diverge. The interview answer proves you read a textbook. The practical answer proves you have been burned by a metric that looked great on a slide and fell apart in production. I want to walk through a handful of common metrics the way I would explain them in an interview, then the way I would actually compute and sanity check them on a real dataset.

The running example throughout is a binary classifier deciding whether a transaction is fraudulent, with a dataset where fraud makes up two percent of cases. That imbalance is not a side note, it is the entire story for several of these metrics.

Accuracy, precision, recall, F1

Interview answer: accuracy is correct predictions over total predictions. It is intuitive and it is the first thing people learn, which is exactly the problem. On our fraud dataset, a model that predicts 'not fraud' for every single transaction scores ninety eight percent accuracy while catching zero fraud. If a hiring interview stops at 'accuracy measures how often the model is right', it has not tested whether you would catch this.

What I actually compute: I never report accuracy alone on an imbalanced problem, full stop. I look at the confusion matrix first, actual counts, not just rates, because rates hide how few positive examples you are testing against. If the test set has two hundred fraud cases out of ten thousand, a recall of ninety percent means one hundred and eighty caught and twenty missed. That is a very different feeling than a bare '0.90' on a dashboard.

Precision and recall trade against each other through the decision threshold, and quoting either one without stating the threshold is close to meaningless. In an interview I would say F1 is the harmonic mean of precision and recall, balancing the two. In practice I rarely use F1 alone because it silently assumes precision and recall matter equally, which is almost never true for a business problem. Missing fraud and annoying a genuine customer with a false alarm usually have very different costs, so I compute precision and recall separately at the threshold the business actually intends to deploy, and I plot the full precision-recall curve so the threshold choice is visible rather than buried.

A concrete worked check: suppose at threshold 0.5 the model flags three hundred transactions, of which one hundred and fifty are genuine fraud and one hundred and fifty are false alarms, while missing fifty real fraud cases. Precision is fifty percent, recall is seventy five percent. F1 comes out around 0.6. None of those three numbers alone tells a fraud team whether this is deployable; the actual conversation is about the cost of one hundred and fifty false alarms against a support team's capacity, which no single scalar metric captures.

AUC, ROC, and the calibration trap

Interview answer: the ROC curve plots true positive rate against false positive rate across all thresholds, and AUC is the area under it, interpreted as the probability the model ranks a random positive example above a random negative one. This is a fine and correct explanation, and it is also the point where I ask a follow up question if I am the one interviewing: does AUC care about class imbalance?

The honest answer is that AUC is fairly robust to imbalance compared with accuracy, because both axes are rates within their own class. But that robustness is also a trap: an AUC of 0.95 on our fraud data can coexist with a model that is practically useless at the threshold you would actually deploy, because AUC averages performance across every possible threshold, including ones nobody would ever use. I have seen AUC quoted as the headline metric in reports where the deployed threshold sits in a region of the curve with mediocre precision, and the reader never finds out.

What I actually compute alongside AUC is precision-recall AUC, which is far more informative under heavy imbalance because it does not have a large negative class quietly propping up the false positive rate. I also check calibration separately, because a model can rank examples perfectly, giving a great AUC, while its predicted probabilities are wildly overconfident or underconfident. If a fraud model outputs a 0.9 probability of fraud and, among all transactions it scores near 0.9, only sixty percent turn out to be fraud, that model is miscalibrated even though its ranking might be excellent. A reliability diagram, bucketing predictions into probability ranges and comparing to observed frequency, is a five minute check that catches this and rarely comes up in interviews at all.

The metric that matters most: how you built the test set

None of the above means anything if the evaluation split leaks information from training into testing. This is the part interviews test least and practice punishes most. If transactions from the same customer appear in both train and test sets, or if you split randomly on a dataset with a strong time component, your precision and recall numbers can look excellent while the model has effectively memorised patterns it will never see again once deployed on genuinely future data.

What I actually do is match the split to how the model will be used. For fraud detection, that almost always means a time based split, training on transactions up to a cutoff date and testing on transactions afterwards, because that mirrors deployment: you are always predicting on the future. I also check for group leakage, making sure no customer or account appears on both sides of the split, since customer level patterns can otherwise be trivially memorised.

My practical takeaway is this: treat the interview definition as the vocabulary, not the answer. When someone asks you about a metric, state the definition quickly, then immediately name the failure mode it hides, the threshold it depends on, and how you would build the split to test it honestly. That is the difference between reciting a glossary and actually knowing whether a model will work when it matters.