Why a probability and not a p-value
A p-value is the probability of seeing a difference at least this large if there were no difference at all. Almost nobody reads it that way. The thing teams want, and the thing they usually believe they have been handed, is the probability that the variant is better, and that quantity does not exist in a significance test. It exists here, and it is an integral over two posterior distributions with no approximation in it.
The model is Beta-Binomial. A conversion rate starts as a Beta distribution expressing what was plausible before the test, each conversion and each miss updates it, and what comes out is another Beta. That is the whole machinery. Every figure on this page is a quadrature over two of them, and nothing on this page samples, so the same counts give the same answer every time you load it.
The expected loss is the number that decides
The probability tells you which way the evidence points. It does not tell you what to do, because it carries no units and no cost.
The expected loss does. It is the average amount of conversion rate you give up by shipping the variant, counted over the worlds where the control was the better of the two. It is measured in rate points, which means a team can set a threshold on it in advance: below this, I do not care which arm wins. That threshold belongs to the business, so this page has no default for it worth having.
Four readings come out of that comparison. Ship it and keep the control are the ones where a single arm’s cost falls under the threshold. Not yet is where neither does. And either is fine is the one worth naming on its own: both costs are under the threshold, which means the test is finished and the answer is that the difference is too small to be worth a rollout. That is a result, and teams routinely mistake it for a failure.
How much disbelief would overturn this
Every Bayesian answer rests on a prior, and a prior is a choice and not a measurement. A tool that hides that is selling certainty it has not got.
So this page asks the question behind the choice: how sceptical would somebody have to be for your conclusion to change? The answer is in observations. A prior centred on the control’s own rate and carrying the weight of so many observations starts from the position that nothing changed, and it has to be argued out of it by the evidence you collected. When the number that would overturn you is smaller than the number your test collected, the conclusion is resting on the prior at least as much as on the data, whatever the probability says.
The table under it shows the same counts under three priors, and the last column is the one to read. When it says the same thing all the way down, the data settled the question. When it does not, the prior was doing the work, and the test did not settle anything.
What this does not license
A Bayesian reading is not permission to watch the dashboard until the number looks good and then stop.
Stopping on an expected loss controls a different quantity than a significance test controls, and the arithmetic here does not care when you looked. It still changes what the figure means. If the plan was a fixed sample, run the fixed sample. If the plan was to stop on a loss threshold, set that threshold before the first visitor arrives and hold to it.
This page reads a test that has finished. The A/B test planner is the one that answers how big it has to be and how long that takes.