What Data Science Success Stories Leave Out

The survivorship bias in every write-up

Read enough data science case studies and a pattern emerges. The team identifies a problem, tries an approach, and within a few paragraphs arrives at a model that beats the baseline by a healthy margin. The narrative arc is clean: problem, method, result, impact. What almost never appears is the fact that the method described was probably the fourth or fifth thing tried, and the first three attempts either failed outright or produced results too weak to be worth a slide.

This is not dishonesty exactly. Write-ups are compressed for an audience that wants the useful part, and nobody wants to read a diary of every dead end. But the compression has a cost. When you only ever see the winning path, you start to believe that competent practitioners go from problem to solution in a straight line. New graduates in particular internalise this and then feel a private sense of failure when their own first three attempts do not work, because the case studies they read never mentioned that this is normal.

I think the more useful framing is that failed attempts are not noise around the real work, they are the real work. The final model is often just the attempt that happened to survive contact with the evaluation set. Everything before it did useful labour: it ruled out hypotheses, exposed data quality problems, and taught the team what the task actually required rather than what the initial brief assumed.

A worked example: churn prediction that almost was not

Imagine a team building a churn model for a subscription product. The first attempt is a gradient boosted tree on account age, usage frequency, and support ticket count, achieving an AUC of 0.81 on a held-out test set. That number looks respectable and would make a fine headline slide. The problem, discovered only after someone checked the timestamps carefully, is that several of the usage features were computed using data collected after the churn event itself. The model was not predicting churn, it was detecting it after the fact, and the 0.81 was an artefact of leakage rather than a genuine signal.

Fixing that leakage and rebuilding the features using only information available at prediction time drops the AUC to 0.63. This is the point in the real project where morale usually dips, and it is also the point that never survives into the final write-up. The team spends the next two weeks trying interaction terms, trying a different target definition based on a rolling thirty-day window instead of a fixed cutoff, and trying to bring in billing data that turns out to be inconsistently logged across regions. None of these attempts individually move the needle much. AUC creeps from 0.63 to 0.65 to 0.67.

The breakthrough, when it comes, is unglamorous: someone notices that the support ticket feature is missing for a third of accounts not because those customers never contacted support but because a data pipeline change silently dropped older tickets. Fixing that single data quality issue, not adding a clever new algorithm, takes the AUC to 0.74. The final report will describe this as feature engineering and imputation handling. It will not describe the two weeks of interaction terms that went nowhere, because those weeks left no trace in the final pipeline, even though they were what forced the team to look closely enough at the data to find the real problem.

Why the missing failures matter in practice

The practical damage from this omission shows up in at least three places. The first is timeline estimation. If stakeholders only ever see success stories that appear to move smoothly from data to deployment, they will underestimate how long a comparable project should take, and they will read a team's slow start as incompetence rather than as the normal cost of understanding a new dataset properly.

The second is evaluation discipline. A failed attempt is often failed precisely because someone checked it properly: they found the leakage, they noticed the target definition was ambiguous, they spotted that the test set overlapped with training data through a shared customer ID. Write-ups that skip the failures also tend to skip the evaluation scrutiny that produced them, leaving readers with a polished number and no sense of how hard that number should be trusted. A reported accuracy of 0.90 means very different things depending on whether it survived three attempts to break it or was accepted on first sight.

The third is a subtler one: false confidence in method choice. If a case study presents a neural network as the solution without mentioning that a well-tuned logistic regression got within two points of it on the same task, readers will overweight the complexity of the winning approach and underweight the value of strong, boring baselines. In the churn example above, a simple logistic regression with the corrected features might get an AUC of 0.71, only three points below the final gradient boosted result. A write-up that omits this comparison implicitly argues for the more complex model on grounds that were never actually tested.

What to do differently

The fix does not require turning every case study into a full lab notebook. It requires a habit: report the baseline that was beaten and briefly name the attempts that did not survive, along with why. A single sentence noting that an earlier version of the model suffered from leakage, and that fixing it changed the reported metric by a meaningful margin, tells a reader far more about how much to trust the final number than the number alone ever could.

As a matter of personal practice, I try to keep a short log of failed configurations alongside the final results, not for publication necessarily, but because it is the fastest way to answer the question someone will eventually ask: why does the model do this and not that. The honest answer is almost always rooted in an earlier failure that taught the team something the final write-up never mentions. Treat that history as part of the evidence, not as an embarrassing footnote, and both your own judgement and anyone reading your work afterwards will be better served for it.