What an MLOps Engineer Actually Does Day to Day

The title undersells the job

When people hear 'MLOps engineer', they usually picture someone wiring up a CI/CD pipeline for models, or clicking deploy on a dashboard. That is part of it, but it is the smallest part. Most of the actual work is closer to plumbing and risk management than to machine learning research. You spend far more time asking 'why did this number change on Tuesday' than you spend training anything new.

The reason the job looks different from its title is that models do not fail the way normal software fails. A web service either returns the wrong status code or it does not. A model can return a perfectly valid, well-formatted prediction that is quietly wrong, and nothing in the system will complain. Finding that kind of failure requires a different mindset: you are not just keeping the lights on, you are keeping the lights honest.

So the day-to-day work sits at the intersection of software engineering discipline and statistical scepticism. You need enough engineering sense to build reliable pipelines, and enough evaluation sense to know when a pipeline is producing garbage that still looks like a sensible number. Neither skill alone is enough; the interesting failures live in the gap between them.

A concrete week: data, retraining, and the quiet decisions

Consider a fraud detection model that scores transactions in real time. On a typical day, a large share of your attention goes to the data feeding it, not the model itself. Suppose the upstream team changes a currency field from a string like '£45.00' to a numeric cents value like 4500. Nothing crashes. The model still returns a score between 0 and 1. But the feature that used to represent transaction size now means something ninety times smaller, and the model's learned weight for that feature is now nonsense. Precision at a fixed threshold might drift from around 0.82 to 0.61 over a few days, and the first sign is usually a spike in customer complaints, not an alert.

Catching that before it reaches customers means building checks that compare the statistical shape of incoming data against a reference distribution, not just checking that a field exists and is numeric. A simple population stability check on that one field, comparing the current week's distribution to last month's, would have flagged the currency change within hours. This is unglamorous work: writing assertions about medians, ranges, and null rates, and deciding what threshold of change should page someone at 2am versus what should just log a warning.

Then there is the retraining decision, which sounds automated but rarely is in practice. Say the model's weekly AUC has dipped from 0.88 to 0.85. Is that seasonal, is it drift, or is it noise from a smaller-than-usual evaluation window? Retraining on the latest data feels like the obvious fix, but if the latest data includes a batch of mislabelled transactions from a support team backlog, you have just baked that mislabelling into the next model version. Part of the job is resisting the urge to retrain reflexively and instead asking whether the labels themselves are trustworthy this week.

Leakage-aware evaluation matters here too. If transactions from the same customer appear in both training and validation splits, the offline AUC will look better than it deserves, sometimes by several points, and the gap only shows up once the model is live and scoring genuinely new customers. A meaningful part of the week is spent auditing splits, checking that time boundaries are respected, and making sure a promising offline number is not just an artefact of the evaluation setup.

Monitoring is a design problem, not a dashboard

Most teams start monitoring by tracking infrastructure metrics: latency, error rate, requests per second. Those matter, but they tell you nothing about whether the model is still doing its job. The harder, more valuable metrics are things like prediction distribution shift, feature drift, and calibration, which is whether a predicted probability of 0.7 actually corresponds to something happening seventy percent of the time.

Building this well means deciding in advance what 'wrong' looks like for this specific model, because generic anomaly detection rarely captures it. For the fraud model, a useful signal might be the weekly fraction of scores above 0.9, tracked against a rolling baseline. If that fraction triples overnight with no corresponding change in actual confirmed fraud, that is a strong hint the feature pipeline broke somewhere, even before any label is available to confirm it. Labels for fraud often arrive weeks later, once disputes are resolved, so you need proxy signals that do not depend on ground truth you will not have for a month.

This is also where the job overlaps with incident response. When something does break, the useful question is never 'what is the model doing wrong' in isolation; it is 'what changed upstream in the last 48 hours', because models rarely break on their own. A postmortem for a model incident tends to read more like a supply chain investigation: which feature source updated, which library version shipped, which upstream table schema changed. Writing these up honestly, including the false alarms and the near misses, is what makes the monitoring system better next time rather than just noisier.

The practical takeaway

If you are trying to understand or break into this role, the useful preparation is not more model architectures. It is getting comfortable with data pipelines that can silently corrupt inputs, evaluation setups that can silently leak information, and monitoring that has to infer problems from proxies because ground truth arrives late or never. The job title says 'operations' but the actual skill being exercised most days is scepticism applied consistently: assuming the number in front of you might be wrong until you have checked how it was produced.

The single habit that pays off most is treating every dashboard metric as a claim that needs a source, not a fact. Before trusting a drop in accuracy or a rise in latency, trace it back to the pipeline stage that produced it. Most of the genuinely hard incidents in production machine learning are not model problems at all; they are data problems wearing a model's clothes.