Monitoring Model Drift: What to Log and When to Retrain

Why silent decay is the real risk

Most people worry about a model breaking loudly: an exception, a null prediction, a service outage. In practice, the more damaging failure mode is quiet. A fraud detection model trained on last year's transaction patterns keeps returning confident scores, the dashboard stays green, and nobody notices that its precision has slipped from 0.91 to 0.78 because the fraud tactics changed. No alarm fires because nothing crashed. The model is simply wrong more often, and wrong in a way that looks like normal variance until someone audits the outcomes.

This is the core reason drift monitoring deserves the same rigour as the original evaluation. When a model ships, we usually have a held-out test set, a clear metric, and a leakage-aware split that gave us honest confidence in the numbers. Once deployed, that discipline tends to evaporate. Teams watch latency and throughput because those are easy to instrument, and they neglect the statistical health of the predictions themselves, because that requires labels, comparison distributions, and a bit more thought.

The fix is not exotic. It is deciding in advance what to log, logging it consistently, and setting explicit thresholds for when a retrain is worth the engineering cost. Without that plan, drift monitoring becomes a reactive exercise: someone notices complaints rising, digs through logs after the fact, and discovers the input distribution shifted three months ago. By then the cost has already been paid.

What to log: inputs, outputs, and the gap between them

There are three distinct things worth tracking, and conflating them is the most common mistake I see. The first is data drift: has the distribution of the input features changed, regardless of whether the model's accuracy has moved yet. The second is concept drift: has the relationship between inputs and the true outcome changed. The third is performance drift: is the model's actual accuracy or error rate declining, which is usually the downstream consequence of the first two but the thing we actually care about.

For data drift, log summary statistics of your input features on a rolling window, and compare them against the training distribution using something like population stability index or a Kolmogorov-Smirnov test for continuous features, and a chi-squared test for categorical ones. Say a credit scoring model was trained when average applicant income was £34,000 with a standard deviation of £9,000. If this month's incoming applicants average £41,000, that is worth flagging even before you see a single wrong prediction, because it tells you the model is now extrapolating into territory it saw rarely during training.

For concept drift, you need labels, which is the hard part. If your outcome arrives with a delay, such as loan default status six months after approval, you cannot know your live accuracy until then. In that case, log proxy signals: the model's confidence distribution, the rate of predictions near the decision boundary, and any available partial feedback like early repayment behaviour. A sudden jump in predictions sitting between 0.45 and 0.55 confidence, when historically only 8% of predictions fell in that band and now 22% do, is a strong hint that the model is becoming less certain about cases it used to handle cleanly.

For performance drift, once labels do arrive, log the full metric you optimised for originally, not a simplified proxy. If the model was tuned for F1 because classes are imbalanced, do not monitor raw accuracy in production, because accuracy can look stable while F1 quietly collapses. Log the metric on a rolling basis with confidence intervals, not a single point estimate, because a small evaluation window will show noisy swings that look like drift but are just sampling variance.

Deciding when to retrain, not just when to worry

Detecting drift and deciding to retrain are separate decisions, and treating them as the same thing leads to either paralysis or wasted effort. A useful habit is to set two thresholds ahead of time: a monitoring threshold that triggers investigation, and a retraining threshold that triggers action, with the gap between them deliberately wide enough to avoid reacting to noise.

Suppose your churn model was validated at 0.83 AUC with a bootstrap confidence interval of roughly plus or minus 0.02. A weekly rolling AUC of 0.81 is well within that noise band and does not justify retraining; it justifies a note and continued watching. A sustained drop to 0.76 over four consecutive weeks, consistently below the lower bound of the original interval, is a genuine signal. The key word is sustained: a single bad week is usually a data quality blip, a marketing campaign that temporarily changed the applicant mix, or a holiday effect, not true drift.

Retraining itself is not free, and that cost should factor into the decision. Every retrain needs a fresh leakage-aware evaluation, not just a swap of the old model for a new one trained on recent data. I would always insist on comparing the retrained model against the current production model on the same recent holdout, using the same metric, before promoting it. It is entirely possible for a retrained model to perform worse than the incumbent if the recent data is noisier or smaller than the original training set, and skipping this check is how teams end up degrading a system while believing they improved it.

Finally, log the retraining decisions themselves, not just the model's predictions. Keep a record of when each retrain happened, what triggered it, what the before-and-after metrics were on a shared holdout, and what changed in the data pipeline. This audit trail is what turns drift monitoring from an ad hoc habit into something reproducible: six months later, when someone asks why the model behaves differently to the version described in an old report, you want an answer that takes two minutes to find rather than an afternoon of archaeology.

The practical takeaway

Treat monitoring as an extension of evaluation, not a separate afterthought bolted on after deployment. Log input distributions and confidence patterns continuously, since these are available immediately and give early warning. Log the true performance metric on a rolling basis with uncertainty estimates the moment labels arrive, and resist the urge to substitute a simpler proxy metric for convenience. Set a monitoring threshold and a stricter, well-separated retraining threshold in advance, so decisions are made from a plan rather than from panic. And whenever you do retrain, validate the new model against the old one on shared, recent, leakage-free data before you trust it. Drift is inevitable; being caught by surprise is not.