Feature Importance Methods: What They Tell You
Importance scores are useful diagnostics, not proof of causal relevance. Here is what they can and cannot support.
Why we reach for importance scores
Whenever I hand over a model, someone asks which features matter most. It is a reasonable question and importance scores give a fast, visual answer: a bar chart ranking variables by how much they seem to drive predictions. The trouble is that this chart answers a narrower question than most people assume. It tells you how much a feature influenced this particular model's output on this particular data, not how important that feature is in the real process generating the data.
That gap matters because importance scores get used for decisions well beyond model debugging: justifying which sensors to keep, which survey questions to drop, which risk factors to flag to a regulator. If the score is quietly answering a different question than the one being asked, those decisions rest on a misunderstanding rather than evidence.
I want to walk through what the common methods actually compute, using a concrete example, and then be specific about where the inference breaks down.
A worked example with correlated features
Suppose I am predicting whether a loan defaults, using two features: years employed and years at current address. In my data these are correlated at around 0.8, since people who stay in a job tend to stay in one place too. I train a random forest and look at built-in gain-based importance. Years employed gets a score of 0.31 and years at current address gets 0.05, even though a simple univariate check shows both features have almost identical individual relationships with default risk.
What happened is not that years employed is five times more informative. It is that the tree-building algorithm picked years employed first at most splits, purely because of how ties were broken and how the greedy search proceeded, and once that feature captured the shared signal there was little unique variance left for the second one to explain. Swap the order the features are presented in, or bootstrap a different sample, and the split could easily go the other way. The importance score is reporting an artefact of correlation and algorithmic tie-breaking, not a stable fact about the world.
Permutation importance, which shuffles a feature and measures the drop in accuracy, has the same vulnerability. If I permute years employed while years at current address is still intact and highly correlated with it, the model can partly reconstruct the missing signal from the correlated feature. The measured importance drop understates the feature's true relevance. Permute both together and the picture changes again. None of these numbers is wrong exactly, they are all faithfully measuring something, but that something depends heavily on which other features happen to be sitting in the dataset.
What SHAP adds and what it still cannot do
SHAP values improve on this by distributing credit more fairly across correlated features using a game-theoretic allocation, and they let you inspect importance at the level of a single prediction rather than only in aggregate. That is a genuine advance for auditing individual decisions: if a loan is denied, I can say that years employed contributed negatively by 0.12 towards the predicted probability of default, holding the rest of the model fixed.
But SHAP values are still conditional on the model you trained and the reference distribution you chose for the baseline. They tell you how the model's output moves when a feature moves, given everything else the model has already learned, including any spurious correlations that survived training. If your training data has a historical bias where postcode correlated with default risk for reasons that had nothing to do with creditworthiness, SHAP will faithfully report postcode as important, because it is important to the model. It cannot tell you that this importance reflects bias rather than a genuine causal driver.
This is the core limitation across every method in this family, whether it is gain, permutation, or SHAP: they describe association within a fitted function, not causal structure in the world. A feature can score highly because it genuinely drives the outcome, because it is a proxy for a hidden cause, because it is correlated with a feature that does drive the outcome, or because of quirks in how the optimiser settled during training. The importance score by itself does not distinguish between these explanations, and no amount of averaging over more permutations will manufacture that distinction from data the method was never designed to use.
Using importance scores without overclaiming
None of this means importance scores are useless, they are genuinely useful for a narrower set of tasks: spotting features the model is leaning on that should raise an ethical or legal flag, sanity-checking that a model has not learned to rely entirely on a leaked feature, and comparing relative reliance across model versions during development. Used for those diagnostic purposes, the caveats above are manageable because you are asking exactly the question the method answers.
Problems start when importance scores get used to argue causal or scientific claims, such as telling a policymaker that a variable is a key driver of an outcome in the real world. For that, you need a study designed for causal inference: randomised experiments where feasible, or techniques like instrumental variables and careful confounding control when they are not. A high SHAP value is not evidence of causal effect, and a low one is not evidence of causal irrelevance, particularly under correlated features.
My practical rule is to always report importance scores alongside a note on correlation structure among the top features, and to be explicit in any write-up about whether the claim is this model relies on this feature or this feature drives the outcome. Those are different sentences and conflating them is where most of the damage happens. Keep the boundary clear and the method remains a genuinely useful diagnostic rather than a source of quietly false confidence.
