What Genetic Perturbation Models Teach Us About Deep Learning

A deceptively simple task

Genetic perturbation models try to answer a concrete question: if you knock out or suppress a particular gene in a cell, what happens to the expression of every other gene? This is measured with single cell sequencing, so for each perturbation you get a high dimensional snapshot of the cell's transcriptional state. The task looks like a natural fit for deep learning. You have rich inputs, rich outputs, and an obvious supervised framing where the model learns a mapping from perturbation identity to expression change.

What makes this domain unusually instructive is that it sits at the intersection of two things researchers usually keep separate: a genuinely hard scientific problem, and a very easy way to fool yourself about how well you have solved it. Gene regulatory networks are highly correlated and modular. Many genes act in pathways, so perturbing one often produces an expression signature that closely resembles the signature of perturbing a related gene. That correlation structure is a gift to any model that wants to look good on a held out test set without doing anything mechanistically interesting.

I find this domain useful to think about precisely because the failure modes are not exotic. They are the same failure modes that show up in NLP leakage through paraphrased test examples, in computer vision leakage through near duplicate images, and in tabular machine learning through target leakage via proxy columns. Perturbation prediction just makes the problem unusually visible, because biologists already have strong priors about which genes are related, so it is easier to catch a model cheating.

The leakage problem, worked through

Suppose you train a model on three thousand single gene knockouts and you split your data randomly into train and test sets. Imagine two genes, call them A and B, that belong to the same signalling pathway and have a correlation of roughly 0.85 in their downstream expression effects across the genome. If A ends up in training and B ends up in test, the model can essentially memorise A's effect profile and output something close to it whenever it sees B, because the two profiles are so similar. On a standard metric like mean squared error against the true expression change, this strategy can produce a respectable score, say an improvement of thirty percent over a naive baseline that just predicts no change at all.

That thirty percent sounds like real signal. The trouble is that it is almost entirely explained by pathway redundancy rather than genuine extrapolation. If you instead split by pathway, holding out entire functional modules so that no test gene has a close relative in training, the same model's advantage over the naive baseline often collapses to something much smaller, sometimes in the single digits. The headline number was real in the sense that it was correctly computed, but it measured interpolation within known biology rather than the thing scientists actually care about: predicting the effect of perturbing a gene whose pathway membership is poorly understood.

This is not a hypothetical concern invented for the sake of argument. It is the direct analogue of a well known problem in drug discovery, where models trained and tested on randomly split chemical libraries perform beautifully until you test on genuinely novel scaffolds, at which point performance often degrades sharply. The lesson generalises: whenever your examples cluster into natural groups with internal similarity, a random split quietly measures how well the model memorises group structure, not how well it reasons about new instances.

Why strong baselines matter here especially

A second lesson from this domain concerns baselines. A common naive baseline predicts the average perturbation effect across all training examples, or even simpler, predicts no change from the unperturbed cell state. These baselines sound almost too simple to be worth reporting, yet in practice they are startlingly hard to beat by a wide margin, because most genes, when perturbed, produce fairly modest and fairly similar downstream shifts; it is only a minority of perturbations that cause dramatic, distinctive rewiring of the transcriptome.

This has a direct consequence for how you should read reported improvements. If a paper reports that a new architecture improves over the mean baseline by, say, five percent on an aggregate metric averaged across thousands of perturbations, that average can be almost entirely driven by the easy majority of perturbations where everything, including the naive baseline, already does reasonably well. The genuinely interesting perturbations, the ones with large and specific effects, might show no improvement at all, or might even be where the fancy model does worst, because those are exactly the examples furthest from anything resembling the training distribution.

The practical fix is to report performance stratified by effect size or by biological novelty, not just as a single pooled number. A model that is excellent on the easy ninety percent and mediocre on the hard ten percent is a very different proposition from one that is uniformly decent, even if their pooled averages happen to match. This stratification habit is cheap to implement and it is one of the clearest signals of whether an evaluation is honest about where the model's competence actually lies.

The practical takeaway

None of this means deep learning has nothing to offer biology. Learned representations of gene expression genuinely do capture structure that simpler linear models miss, and there are real cases where a neural network extrapolates sensibly to unseen perturbations because it has learned something closer to a mechanistic embedding of gene function. The point is that you only find out which situation you are in by being deliberate about the split.

If I had to compress this into one habit worth carrying into any machine learning project, biological or not, it would be this: before trusting a headline number, ask what grouping structure exists in your data and whether your split respects it. Hold out by cluster, by pathway, by scaffold, by source, by whatever the natural correlated unit is in your domain, and always compare against a naive baseline that is embarrassingly simple to compute. If the gap over that baseline survives a split designed to be adversarial to memorisation, you have learned something real. If it does not, you have learned something about your correlation structure, which is useful too, just not the thing the paper probably claimed.