The model is one line
A problem that one user in p runs into is missed by all n participants with probability one minus p, raised to the power n. So the chance of seeing it at least once is one minus that. Everything on this page is that formula read in one direction or the other.
It is a good model of one thing and a poor model of several others, and the last section says which.
Where “five users” comes from
The claim is not invented and it is not wrong. It rests on a problem frequency of about 0.31: if roughly a third of users hit a problem, five participants find 85 percent of such problems. That is the arithmetic, and at that frequency five is a sensible number.
The assumption is doing all the work, and it travels without the assumption attached.
A product in its first month has problems that common. A product that has been in the market for a year does not, because the problems a third of people hit were found in the first week and fixed in the second. What is left is the kind one user in ten meets, or one in twenty, and the table above shows what five participants do with those: 41 percent, and 23 percent.
Five participants is not a number of participants. It is an answer to a question about problem frequency, and the question gets dropped.
The average is not the result
Eighty-five percent is an average over repeats of the study. You run the study once.
The second block above works out the range the one run you do would land in. The arithmetic is the same model: if each problem is found or missed independently with the same probability, the number found follows a binomial distribution, and its central range is the spread across repeats. It is derived here from the same model, so you can change the inputs and watch it move.
The range is wider than most people expect, and it is widest exactly where the claim is strongest. That is the honest reason to run more sessions: not that the average rises much, but that the range you are gambling on narrows.
What to do with this
Decide the frequency you care about before you decide the number of participants, and write it down. A team that says “problems at least one in ten of our users would hit” has made a decision somebody can disagree with, which is more than “we will test with five” offers.
Then run in rounds. Five, fix, five more. The formula above assumes one study and independent draws; two rounds of five with fixes in between is a different and usually better design, because the second five meet a product the first five already improved.
And recruit the second five from somewhere else. The model’s weakest assumption is that participants are independent draws, and five people from the same customer-success list are not five independent draws of anything.
What this does not model
Every problem has the same probability in this model, and they do not. Severity does not appear at all: finding 85 percent of problems is not finding 85 percent of the pain, and a study that misses the one problem that loses the sale has done badly by any measure this page can compute.
Participants are assumed independent. They are not, if they share a role, a tenure or a recruiting channel.
And the tasks set the result more than the sample size does. Five participants on the right task beat fifteen on the wrong one, and no amount of arithmetic here will tell you which task that is.