Cross-Validation Folds for Grouped Data: Silent Leakage

The assumption nobody checks

Cross-validation works by pretending each fold is a fresh, unseen sample of the population you care about. That pretence relies on one quiet assumption: rows are independent. In practice, a huge amount of real data is not independent at all. It arrives in groups: multiple scans from the same patient, multiple posts from the same user, multiple sensor readings from the same machine, multiple sentences from the same document. Standard k-fold cross-validation shuffles rows randomly and does not care whether two nearly identical rows end up in different folds.

This matters because a model does not need to learn the general pattern you want if it can instead learn the specific identity of a group. If patient A appears in both the training fold and the test fold, the model can partly memorise patient A's quirks rather than learning what generalises to patient Z, who it has never seen. The test score still looks respectable, sometimes excellent, but it is measuring something closer to recall of familiar entities than genuine predictive skill.

The leak is silent because nothing in the pipeline throws an error. Your cross-validation loop runs, folds are stratified nicely, the code executes without warnings. The only symptom is a gap between validation performance and real-world performance on genuinely new groups, and that gap only shows up later, often after deployment, when it is expensive to discover.

A worked example

Suppose you are building a classifier to detect a medical condition from repeated scans, and you have 200 patients, each contributing five scans, so 1,000 rows total. You run standard five-fold cross-validation on the 1,000 rows, shuffled randomly. Because each patient's five scans are similar to one another, roughly four of each patient's five scans will land in the training folds whenever that patient's data is split, with only one or two held out in the test fold in any given split.

The model effectively gets to see most of each patient before being tested on the rest of that same patient. If patients have any idiosyncratic signal, a distinctive baseline noise pattern, a device artefact, subtle anatomical traits, the model can exploit that instead of the disease signal you actually want it to learn. Say this pushes reported accuracy to around 94%. It looks strong. But when the model is deployed on new patients it has never encountered, accuracy could plausibly fall to somewhere in the 70s, because the shortcut it learned, patient identity, no longer helps.

Now redo the split using group k-fold, where all five scans from a given patient are forced into the same fold, never split across train and test. With 200 patients, each fold contains roughly 40 entirely new patients. The model can no longer lean on patient-specific quirks, because every test patient is genuinely unseen. Accuracy drops, honestly, to something closer to what you would actually see in deployment, perhaps the 70s I mentioned above. That number is less flattering, but it is the number that matters, because it is the number you can trust when someone asks whether the model will work on the next patient who walks through the door.

Where this shows up beyond medicine

The same pattern recurs anywhere data is naturally clustered. In text classification, if a corpus contains multiple paragraphs pulled from the same document, random splitting lets a model partially memorise document-specific vocabulary and style rather than learning the general category. In recommendation and behavioural modelling, multiple interactions from the same user leaking across train and test let a model learn that user's preferences rather than transferable patterns. In time series with repeated measurements from the same sensor or subject, the same risk appears, compounded further by temporal leakage if future rows sit in training folds and past rows sit in test folds.

It is worth being precise about what counts as a group. Sometimes the grouping variable is obvious, a patient ID or user ID sitting right there in the data. Sometimes it is not explicit and needs to be constructed, for example grouping by recording session, by geographic site, or by the annotator who labelled the data. If different annotators have systematically different labelling habits, and your split does not account for that, you can leak annotator identity instead of task signal. The general rule is to ask: what is the unit that will be genuinely new at deployment time? Then group by that unit, however it is defined, not by row.

The fix, in most tooling, is a small change: swap a standard k-fold splitter for a group-aware one that takes a group label and guarantees no group crosses the train and test boundary. It costs almost nothing to implement and should be treated as a default checklist item, not an optional refinement applied only when leakage is suspected.

Practical takeaway

Before running any cross-validation, ask a simple question: could the same underlying entity, patient, user, document, session, sensor, generate more than one row in this dataset? If the answer is yes, random k-fold is the wrong tool, because it will let the model partially identify entities rather than purely learn the relationship you actually care about, and your validation score will be optimistic in a way that is invisible until deployment.

The practical fix is grouped splitting: keep every row from a given entity entirely within one fold, whether that is achieved through group k-fold, leave-one-group-out, or a manually constructed split by patient, user, or session ID. Expect the resulting scores to be lower than what random splitting would have shown you. That drop is not a failure of the model; it is the removal of a false signal you were never entitled to trust. A lower, honest number that predicts real-world performance is worth far more than a higher number that quietly measures memorisation.