Threat Modelling for ML Systems, Not Just Accuracy

Accuracy tells you almost nothing about safety

Suppose I build a classifier for detecting fraudulent transactions and it scores 97% accuracy on a held-out test set. That number sounds impressive until I ask what the base rate of fraud actually is. If only 2% of transactions are fraudulent, a model that predicts 'not fraud' for everything scores 98% accuracy while catching precisely zero fraud. This is not a contrived edge case; it is the default shape of most real-world classification problems, where the interesting class is rare and the cost of missing it is wildly asymmetric to the cost of a false alarm.

The deeper problem is that accuracy, and most single-number metrics, compress every kind of error into one undifferentiated bucket. A missed cancer diagnosis and a missed spam email get treated as equivalent mistakes if you only look at overall correctness. Threat modelling is the discipline of refusing that compression. Instead of asking 'how often is this model right', you ask 'when this model is wrong, who is harmed, how badly, and under what conditions does that wrongness become likely'. That is a fundamentally different question, and it changes what you build, test, and monitor.

I think of threat modelling for ML as borrowing the mindset of security engineering rather than statistics. A security engineer does not ask whether a system usually resists attack; they ask what happens under the worst plausible input, deliberately constructed or otherwise. Applied to ML, this means enumerating the ways a model's assumptions can be violated in deployment: shifted input distributions, adversarial manipulation, rare subgroups, feedback loops, and silent data drift. None of these show up in a standard accuracy figure, yet all of them are common causes of real incidents.

A worked example: the failure modes hiding behind one number

Take a concrete case. Imagine a model for triaging support tickets, trained on a year of historical data, achieving 91% accuracy at classifying urgency. That single figure hides several distinct threats worth separating out individually.

• Subgroup failure: the model might be 96% accurate on tickets written in fluent English and 74% accurate on tickets containing typos, slang, or a second language mixed in. Overall accuracy averages these away, but the second group is exactly where urgent problems from frustrated, non-native-speaking users are most likely to be misclassified as low priority.
• Temporal drift: the training data covers a year with no major product outage. If a new feature launches and generates an unfamiliar cluster of complaints, the model has never seen that vocabulary and may default to whatever class was most common in training, silently misrouting a spike of genuinely urgent tickets during the exact period when accuracy matters most.
• Feedback contamination: if the model's own past predictions influenced how tickets were labelled or routed historically (for instance, tickets it marked low priority were reviewed less carefully and thus less likely to be relabelled as urgent), the training data itself encodes the model's prior mistakes, and retraining on it can entrench the same blind spot rather than correcting it.
• Adversarial or gaming behaviour: if users learn that certain keywords trigger faster response, they may start inserting them regardless of true urgency, degrading the signal the model was trained to rely on.

Each of these is invisible to a single accuracy number computed on an i.i.d. test split, and each requires a different mitigation: stratified evaluation for the first, a monitored retraining or alerting policy for the second, careful audit of label provenance for the third, and ongoing feature stability checks for the fourth. Notice that none of these are exotic; they are the ordinary, boring ways deployed systems degrade.

Building the threat model: questions before metrics

A practical threat model for an ML system does not start with a metric. It starts with a small set of structured questions, asked before a single model is trained. First: what does a false negative cost, and to whom, compared with a false positive? Writing this down explicitly, even roughly, forces you to design the evaluation around the asymmetry rather than discovering it after deployment when the cost has already been paid by a real person or a real budget.

Second: what subpopulations exist in the data, and does performance hold across them? This means deliberately slicing evaluation by known covariates, not waiting for a complaint to reveal the slice that matters. If ticket language, device type, or geography plausibly correlates with error rate, evaluate on those slices explicitly rather than hoping the aggregate number is representative of every group inside it.

Third: what happens when the input distribution shifts, and how would you know? This is where leakage-aware evaluation becomes essential. A test set drawn from the same time window as training data will always look healthier than reality, because it cannot capture drift that has not happened yet. A more honest evaluation holds out the most recent slice of data chronologically, simulating the actual condition the model will face: predicting on data it has never conceptually seen, generated after the model was frozen.

Fourth: is there a strong, boring baseline that this model actually needs to beat, and by how much, given the cost structure from the first question? A model that edges out a simple rule-based baseline by half a percentage point of accuracy, while being far less interpretable and harder to audit for the specific failure modes above, may not be worth deploying at all. Threat modelling and baseline comparison are two sides of the same discipline: both refuse to let a single flattering number end the conversation.

The practical takeaway

None of this requires exotic tooling. It requires discipline in how you frame evaluation before you start, and honesty about what a single aggregate metric cannot tell you. Concretely: define the cost asymmetry of your errors in writing, slice your evaluation by every subgroup you can plausibly identify, test against a time-based or otherwise leakage-aware split that mimics real deployment conditions, and always compare against a baseline strong enough to be genuinely humbling.

Threat modelling is not a replacement for accuracy, precision, or recall; it is the frame that tells you which of those numbers actually matter, and for whom. The models that fail badly in production are rarely the ones with a mediocre accuracy score. They are the ones where a good aggregate score concealed a specific, foreseeable failure that nobody bothered to look for.