SMOTE and Synthetic Data: When It Backfires
Oversampling can rescue a model from a severe class imbalance, or it can quietly poison your evaluation. The difference lies almost entirely in where you apply it.
The appeal, and the trap
Class imbalance is one of the first walls every practitioner hits. You have a fraud detection dataset with a handful of positives per thousand transactions, or a medical dataset where the condition of interest appears in three percent of cases. A model trained naively on this will often just predict the majority class and still post a respectable accuracy figure. SMOTE, the Synthetic Minority Oversampling Technique, offers a tempting fix: generate new synthetic minority examples by interpolating between existing minority points and their nearest neighbours, then rebalance the training set before fitting a classifier.
The trap is that SMOTE is frequently applied in a way that makes results look better without making the model actually better. The most common failure is applying oversampling before splitting into train and test, or before cross validation folds are created. If you generate synthetic minority samples from the full dataset and then split, synthetic points derived from a test-set example can end up in your training fold, and the near-identical original can end up in the test fold. Your model isn't learning to generalise; it's partially memorising interpolated copies of the very data it will be scored against.
This isn't a hypothetical edge case. It's one of the most reproducible ways to inflate a paper's or a portfolio project's reported F1 score. I've seen the same dataset produce a reported recall in the high nineties with leaky SMOTE and a much more modest, and far more honest, recall once oversampling was moved strictly inside the cross validation loop, fitted only on the training fold each time.
A worked example of the leak
Suppose you have 1,000 examples, 950 negative and 50 positive. You apply SMOTE to the whole dataset, generating 900 synthetic positive examples so the classes are balanced at 950 versus 950. Now you do a standard 80/20 train-test split on this augmented set of 1,900 rows. Because SMOTE interpolates between a real minority point and its nearest minority neighbours, many synthetic points sit extremely close in feature space to the real points they were derived from. Some of those real points, and some of their synthetic near-duplicates, will land on opposite sides of your split by pure chance.
The test set now contains points that are almost identical to training examples. Your classifier, particularly a nearest-neighbour-based one or a tree ensemble that partitions feature space finely, doesn't need to learn the true minority-class boundary at all. It just needs to recognise the local neighbourhood, which it has effectively already seen. The evaluation metric rewards this recognition as if it were generalisation. When the model later meets genuinely new fraud patterns or genuinely new patient presentations, none of which resemble the interpolated training neighbourhoods, performance collapses.
The fix is mechanical but non-negotiable: oversampling belongs strictly inside each training fold, refitted from scratch every time, using only the labels and features available in that fold. In practice this means SMOTE goes inside a pipeline object alongside the classifier, not as a preprocessing step applied once to the whole dataset. Any library that supports this properly will let you chain the resampler and the estimator so that cross validation calls fit_resample only on the training portion of each fold.
Beyond leakage: distortion of the decision boundary
Even when SMOTE is applied correctly, with no leakage, it can still backfire for a subtler reason: it assumes the minority class is locally smooth and that interpolating between neighbours produces plausible new examples. That assumption holds reasonably well for dense, well-separated clusters. It breaks down for minority classes that are themselves noisy, overlapping with the majority class, or scattered across several disconnected subclusters.
Consider a minority class formed of two distinct subpopulations, say early-stage and late-stage presentations of a rare condition, sitting in different regions of feature space with a gap of majority-class points between them. SMOTE's nearest-neighbour interpolation can generate synthetic points that fall inside that gap, effectively inventing minority examples that look like blends of majority and minority characteristics. The classifier then learns a decision boundary shaped by these fabricated in-between cases, which don't correspond to anything that occurs in reality. Precision on real minority cases can actually fall even as the reported balanced accuracy on synthetic-augmented validation data looks fine.
There's also a scale problem. If your minority class has only twenty or thirty examples, SMOTE's nearest-neighbour calculation is working with an extremely thin sample of the true distribution. Every synthetic point is an interpolation of interpolations after enough oversampling, and the effective diversity of your training set doesn't grow the way the row count suggests. You end up with more rows but not more information, and some classifiers, especially those sensitive to local density like k-nearest neighbours, are particularly easy to fool this way.
What actually works, and the honest takeaway
None of this means SMOTE is a bad idea; it means SMOTE is a tool with specific assumptions that need checking, not a default switch to flip whenever you see an imbalanced label column. Before reaching for it, I'd rather try class weighting in the loss function, which changes how errors are penalised without inventing any new rows at all, and compare it honestly against SMOTE using the same nested cross validation setup. Often the weighted model matches or beats the oversampled one, with far less risk of leakage or boundary distortion.
If you do use SMOTE, treat it as part of the model, not part of the data. Put it inside the pipeline, refit it per fold, and evaluate on metrics that actually reflect the cost of your errors, such as precision-recall curves or a cost-weighted score, rather than accuracy or a naively computed F1 that can be gamed by near-duplicate synthetic points. Inspect a sample of the generated synthetic examples directly; if they look implausible or sit in regions your domain knowledge says shouldn't contain minority cases, that's a signal the interpolation assumption has failed for your data.
The broader lesson generalises well past SMOTE. Any technique that manufactures data, whether it's oversampling, data augmentation for images, or generative upsampling, must sit downstream of your train-test split, never upstream of it. The moment synthetic generation touches information from outside the training fold, your evaluation stops measuring generalisation and starts measuring memorisation, and no amount of algorithmic sophistication in the classifier afterwards will save you from that.