A Worked Example of Bayesian Model Comparison
Instead of asking whether a p-value crossed 0.05, I show how to ask a more useful question: how much more should I believe in one model over another, given the data.
Why a single p-value undersells the question
A p-value answers a narrow question: if the null hypothesis were exactly true, how surprising would data this extreme (or more extreme) be. It says nothing about how plausible the alternative hypothesis actually is, and it says nothing about the size or practical relevance of an effect. Two experiments can produce the same p-value of 0.03 while telling completely different stories about the underlying effect, depending on sample size and the actual magnitude involved.
What I actually want to know, most of the time, is something closer to: given what I observed, how much more should I believe model A over model B. That is a comparison between two explanations of the data, weighted by how well each one predicts what actually happened. Bayesian model comparison, through the Bayes factor, answers exactly that question, and it does so on a continuous scale rather than a binary reject-or-not decision at an arbitrary threshold.
This matters practically because thresholded decisions hide information. A p-value of 0.049 and a p-value of 0.051 lead to opposite headlines under the usual 0.05 convention, even though the underlying evidence is nearly identical. A Bayes factor does not force that cliff-edge behaviour; it just reports a ratio of evidence, and you decide how much ratio is enough for your purposes.
Setting up a concrete comparison
Suppose I am testing whether a new feature in a classifier improves accuracy on a held-out, leakage-free test set of 200 examples. The baseline model gets 120 correct out of 200. The new model gets 132 correct out of 200. A quick two-proportion test might return a p-value around 0.09, which under the usual convention would be reported as 'not significant', and the new feature would likely be dropped.
Now consider the same data through model comparison. I define two models of how the improved accuracy arose. Model B0 says the true accuracy of the new model equals the baseline's true accuracy, roughly 0.60, with no improvement. Model B1 says the true accuracy is unknown and could plausibly be anywhere in a sensible range, say I place a Beta(6,4) prior over it, reflecting genuine uncertainty but centred a bit above chance, consistent with what a reasonable practitioner might believe before seeing this result.
To compare them, I compute the marginal likelihood of the observed 132 successes out of 200 trials under each model. Under B0, this is just the binomial probability of 132 successes given a fixed probability of 0.60, which comes out fairly small because 132/200 = 0.66 is somewhat away from 0.60. Under B1, I integrate the binomial likelihood over the Beta(6,4) prior, which is analytically tractable and produces a Beta-Binomial marginal likelihood. Because B1 allows the true rate to shift towards 0.66 rather than being pinned at 0.60, its marginal likelihood is noticeably larger.
Dividing the two marginal likelihoods gives the Bayes factor, and in a case like this the ratio typically lands somewhere in the range of 2 to 4 in favour of B1. That is worth stating plainly: this is modest evidence for an improvement, not strong evidence, and definitely not proof. But it is a more honest and more granular statement than 'p equals 0.09, therefore no effect', because it tells me the improvement is more likely real than not, just not by an overwhelming margin.
Reading the Bayes factor honestly
A Bayes factor of 3 means the data are three times more likely under the model that assumes an improvement than under the model that assumes none. That is useful information for deciding what to do next, but it is not a licence to declare victory. Conventional guidance treats Bayes factors below 3 as barely worth mentioning, values between 3 and 10 as moderate evidence, and values above 10 as strong evidence. A result of 3 sits right at the boundary of being interesting, which is itself an honest and informative place to land, unlike a p-value that forces a binary verdict.
It is also worth being explicit that the Bayes factor depends on the prior I chose for B1. If I had instead used a much wider, flatter prior that spread belief thinly across the whole 0 to 1 range, the marginal likelihood for B1 would drop, because that prior wastes probability mass on accuracy values far from what was actually observed, and the resulting Bayes factor would shrink. This sensitivity is not a flaw to hide; it is a feature to report. Stating the prior alongside the result, and ideally showing how the conclusion changes under a couple of reasonable alternative priors, is part of doing this properly. A Bayes factor without a stated prior is about as informative as a p-value without a stated sample size.
Where this really earns its keep is in comparing more than two models at once, something a p-value struggles to do gracefully. If I have a baseline, a model with one new feature, and a model with two new features, I can compute marginal likelihoods for all three and directly rank them, then normalise into posterior model probabilities that sum to one. That gives me a distribution over 'which explanation is best' rather than a sequence of pairwise significance tests that quietly inflate the false positive rate the more comparisons I run.
The practical takeaway
None of this requires abandoning frequentist tools outright; p-values still have a place, particularly when the goal is strict error-rate control across many repeated decisions. But when the goal is to reason about which explanation the evidence actually favours, and by how much, Bayesian model comparison gives a sharper and more honest answer. It replaces a single threshold-crossing verdict with a ratio that reflects the actual weight of evidence, and it forces the prior assumptions into the open where they can be scrutinised rather than smuggled in silently.
My practical rule is this: whenever I catch myself about to report only a p-value for a model comparison, I try to also compute, or at least sketch, the corresponding Bayes factor under a couple of sensible priors. If the two methods agree, that agreement is reassuring. If they disagree, the disagreement is usually the most interesting and honest part of the whole analysis, and it is exactly the sort of thing a single number at the 0.05 line would have hidden from view.
