SHAP, LIME, and When Neither Method Is Honest

Why the question matters more than the method

Every time I see a feature attribution plot presented as ground truth about how a model reasons, I get a little nervous. SHAP and LIME are both genuinely useful, but they are answers to specific mathematical questions, not windows into a model's mind. The trouble starts when people forget which question was asked. SHAP asks: if we imagine all possible subsets of features being present or absent, how much does each feature contribute to moving the prediction away from a baseline, on average, across those subsets? LIME asks a different question: if I perturb this one instance a bit and fit a simple, interpretable model locally around it, what does that local surrogate say matters?

These are not the same question, and they are not guaranteed to agree, because they are built on different assumptions about what counts as a faithful local description of a complicated, non-linear function. A model can have genuinely non-linear, interacting behaviour near a data point, and both methods will produce a tidy bar chart. Neither chart is a lie in the sense of being computed wrongly. But a tidy chart implies a simplicity that the underlying model may not have, and that gap is where honest interpretability quietly breaks down.

What SHAP and LIME actually compute

SHAP is grounded in cooperative game theory. Treat each feature as a player contributing to a payoff, the model's prediction, and Shapley values distribute that payoff fairly across players based on their average marginal contribution over all possible coalitions. This gives SHAP some appealing properties: attributions sum exactly to the difference between the prediction and a baseline, and features that contribute nothing get exactly zero. The cost is computational. Exact Shapley values require evaluating the model on an exponential number of feature subsets, so in practice we use approximations, sampling-based estimators for general models, or faster exact algorithms for tree ensembles. Those approximations introduce their own variance, which is rarely reported alongside the pretty summary plot.

LIME takes a more pragmatic route. It samples points around the instance you care about, weights them by proximity, gets the black-box model's predictions on those samples, and fits a simple linear model to that local neighbourhood. The coefficients of that linear model become your explanation. This is fast and flexible, and it can be applied to almost any model, including ones where SHAP approximations are awkward. But the explanation depends heavily on choices that are rarely surfaced: how you define the neighbourhood, how many samples you draw, which kernel width you use for weighting distance. Change the kernel width and you can change which features look important, sometimes substantially, without touching the model at all.

Here is a concrete illustration. Suppose a credit scoring model has learned a threshold effect: income above a certain level strongly reduces predicted default risk, but only once a second feature, existing debt, is below a certain level; otherwise income barely matters. For an applicant sitting near both thresholds, LIME's local linear fit might attribute most of the effect to income, because the sampled neighbourhood happened to vary income more usefully than debt. SHAP, averaging over many coalitions, is more likely to split credit between both features, since it explicitly accounts for their interaction across many combinations. Run LIME twice with different random seeds on the exact same instance and you may get noticeably different rankings. Run SHAP with a small sample size for approximation and you get the same instability, just less visibly.

Where neither method is honest

The deepest problem is not sampling noise, it is the baseline. SHAP's attributions are always relative to a reference point, often the average feature values across a background dataset, or an expected value under some assumed feature distribution. Change that baseline and the attributions change, sometimes dramatically, even though the model itself has not moved at all. If your background dataset does not resemble the population the model actually sees in production, your explanations describe a comparison that never occurs in the real world. This is a leakage-adjacent trap in disguise: an explanation built on an unrepresentative reference set can look rigorous while quietly encoding an assumption nobody checked.

Correlated features make this worse for both methods. If two features are highly correlated, perturbing one while holding the other fixed, which both SHAP's coalition sampling and LIME's neighbourhood sampling effectively do, creates synthetic data points that never occur naturally. Imagine a model using both age and years of work experience, which are tightly correlated in most populations. Perturbing age alone while pinning experience produces a twenty-year-old with thirty years of experience. The model was never trained to see that combination, so its prediction there is essentially extrapolation, and the resulting attribution reflects the model's behaviour in a region it has never sensibly learned. The explanation is computed correctly and still describes something that cannot happen.

There is also the issue of what counts as a satisfying explanation versus a true one. Both methods will happily produce a clean, confident-looking chart for a model that is badly miscalibrated, poorly validated, or trained on leaked features. A feature that looks important because it leaked target information during training will show up as important in SHAP and LIME alike, and the neat visualisation gives that leakage an unearned air of legitimacy. Interpretability tools explain the model you built, not whether you should trust it, and conflating the two is one of the most common mistakes I see.

A practical way to use both honestly

None of this means abandon SHAP or LIME. It means using them the way you would use any point estimate: with an explicit statement of assumptions and some sense of variance. Report the background dataset used for SHAP baselines and check it matches the deployment population. Run LIME multiple times with different seeds and kernel widths and check whether the top features are stable, not just present once. Where features are correlated, consider grouping them before attribution rather than pretending they vary independently.

Most usefully, treat agreement between SHAP and LIME as a signal worth investigating rather than a box to tick. When they broadly agree, you have modest evidence the explanation reflects something real about locally linear structure in the model. When they diverge, that divergence is informative: it usually means the model's behaviour near that instance is genuinely non-linear or interaction-heavy, which is worth knowing regardless of which chart you print in the report. An honest explanation admits its own baseline, its own sampling noise, and the specific question it was built to answer, rather than presenting a single confident bar chart as the model's mind laid bare.