The ML Engineer Skill Set: Modelling Is the Smaller Half

Why this surprises people

When I started studying machine learning properly, I assumed the job was mostly about choosing the right architecture and tuning it well. That assumption survives about one real project. In practice, calling fit on a model is often an afternoon's work once the data is ready. The weeks before and after that call are where the actual engineering happens: sourcing and cleaning data, building a split that does not lie to you, defining a metric that reflects the real cost of errors, and then watching the model in production as the world quietly shifts underneath it.

This is not a fashionable opinion to hold, because modelling is the part that gets taught, written about, and shown off in tutorials. It is genuinely interesting: there is elegant maths behind gradient descent, and a satisfying moment when a loss curve finally bends the right way. But interesting and important are not the same thing. A mediocre model trained on a clean, well-understood dataset with a trustworthy evaluation protocol will beat a brilliant model trained on leaky, mislabelled data almost every time it matters, which is in production, not in a notebook.

A worked example: the leak you do not see

Suppose you are building a model to predict whether a customer will cancel a subscription in the next thirty days, using eighteen months of historical account data. You engineer features: number of support tickets, days since last login, whether the last invoice failed. You do a standard random train test split, 80/20, and get an impressive area under the curve of around 0.93. That number feels great. It is also almost certainly wrong in a way that will hurt you later.

The problem is that a random split scatters rows from the same customer's timeline across both train and test sets, and it lets the model see information from after the prediction point, because features like total lifetime support tickets are computed over the whole history, including the weeks right before cancellation. The model is not learning to forecast churn; it is learning to recognise the fingerprints of a cancellation that has already effectively happened. In production, where you only have data up to today, that 0.93 collapses, sometimes to something only modestly better than a simple rule based on days since last login.

The fix is unglamorous: split by time and by customer, so that every training example is computed strictly using information available before the cutoff date, and no customer appears in both train and test. Once you do that properly, a realistic AUC might land closer to 0.78. That is a worse number on paper and a far more useful model in practice, because it tells you the truth about what the system can actually do once deployed. Getting to that honest 0.78 took far more thought than the model architecture did; the gradient boosted tree or logistic regression you choose at the end barely moves the needle compared with getting the split and the feature timing right.

The parts that actually consume the calendar

If you map out where the hours actually go on a typical project, a rough and honest picture looks like this: understanding the business problem and translating it into a measurable target, perhaps a week; sourcing, joining, and cleaning data from systems that were never designed to talk to each other, several weeks; building a leakage-aware evaluation harness and defining the right metric, days that feel like they should take hours; the modelling step itself, often a few days of trying two or three sensible baselines and one more sophisticated approach; and then, if the project is real rather than a demo, an open-ended commitment to monitoring, retraining, and explaining the model to people who did not build it.

• Data quality work: deduplication, handling missing values honestly rather than silently, reconciling inconsistent labels across time or across annotators.
• Evaluation design: choosing a split that matches how the model will actually be used, picking metrics that reflect asymmetric costs, and reporting uncertainty rather than a single point estimate.
• Pipeline and infrastructure: making feature computation reproducible between training and serving, so the numbers a model was trained on match what it sees live.
• Monitoring after deployment: tracking input drift, output distributions, and downstream business metrics, because a model that was accurate in January can degrade quietly by June without a single line of code changing.

None of these tasks require inventing a new loss function or reading the latest architecture paper. They require the kind of patience that treats a pandas dataframe with the same scepticism you would apply to a stranger's claim. That scepticism is the actual differentiator between engineers whose models survive contact with production and those whose models look excellent in a slide deck and quietly get switched off six months later.

The practical takeaway

None of this is an argument against learning modelling well. You still need to understand bias and variance, regularisation, and why a particular architecture suits a particular kind of data. But it is an argument for sequencing your effort correctly. Before you tune a single hyperparameter, ask whether your split respects time and grouping, whether your metric reflects the actual cost of the mistakes you will make, and whether the features you are computing in training will be computable, identically, at serving time.

A useful habit is to write down, before touching a model, exactly how you will know if it has failed: what baseline it must beat, on what slice of data, measured by what metric, and evaluated on a split that could not have leaked information from the future. If you cannot answer that clearly, the choice of algorithm is the least of your problems, and no amount of modelling skill will rescue an evaluation that was quietly rigged from the start.