Skip navigation
Nico Santangelo
Get in touch
Instruments

Is that difference real?

A SUS, UMUX-Lite, CSAT or NPS score with the interval around it, and a straight answer on whether the study can tell it apart from the figure you are comparing it with.

What you measured

The score

74.0

  • What the study supports63.2 to 84.8
  • Width of the interval21.6
  • People12

A score is a point and the interval is what the study can support. Two studies with the same score and different sample sizes are not the same result, and the number on its own cannot show that.

The score, and the range around it
The score, and the range around itA point on the metric scale with a bar showing the range the study supports, and a line at the figure it is being compared with.63850255075100Compared with 68

The interval crosses the line, so this study cannot tell the two apart. That is a finding, and it is the one to report.

What a narrower answer would cost
People it would take

57

At the spread you entered. The multiplier is held at the value for the sample you have, so for a small study this is a slight underestimate.

Summary to paste into a ticket

The score: 74.0 (n 12, SUS)
What the study supports: 63.2 to 84.8
Compared with 68: The interval crosses the line, so this study cannot tell the two apart. That is a finding, and it is the one to report.

https://youex.de/tools/ux-metric-scorer/

Why a score needs an interval

A survey score is an estimate built from the people who happened to answer. Run the same study again with twelve different participants and you get a different number. The interval is how different that number could reasonably have been, and without it a score is a claim with no evidence attached.

The worked example on this page is the case that matters. Twelve people, a SUS of 74, a spread of 17. Seventy-four looks like a comfortable win over the figure usually quoted as an average, and the interval runs from 63 to 85. The study is compatible with a system well below average and with one well above it. The honest report is that twelve people could not tell.

That is not a reason to stop measuring. It is a reason to say what the measurement supports, and to know what it would take to support more. The last block on the page works that backwards: at a spread of 17, cutting the interval to ten points takes about 56 responses.

How each one is computed

SUS and UMUX-Lite are means. Enter the score and the spread across respondents and the interval is the standard one for a mean, using the t distribution in place of the normal one. At twelve responses the t multiplier is about twelve percent larger than z, so an interval built on z is about twelve percent too narrow. Most calculators use z.

CSAT is a proportion: the share who picked one of the boxes you count as satisfied. Its interval is the adjusted Wald interval and not the textbook one, because satisfaction results sit near the top of the scale and the textbook interval starts reporting percentages above a hundred exactly there. The survey sample size page has the longer version of that argument.

NPS is the one most often got wrong. It is the share of promoters minus the share of detractors, taken from the same sample, so the interval a proportion calculator gives for it is the wrong interval. Score each response as plus one, zero or minus one and the promoter score is a mean, its spread follows from the three counts, and the same arithmetic as SUS applies. The width that comes out surprises people: a hundred responses carry an interval around thirty points wide, which is wider than most of the quarter-on-quarter movements that get reported as changes.

The spread is the input nobody records

For SUS and UMUX-Lite the interval needs a standard deviation, and almost nobody writes one down. The mean goes in the deck and the spread goes in the bin.

If you have the raw responses, compute it; any spreadsheet will. If you do not, the honest move is to try a range: the interval at a spread of 12 and at a spread of 22 brackets most of what you are likely to have had, and if your conclusion survives both, the missing number did not matter. If the conclusion flips, you have learned that your report rests on a figure you did not keep.

What to compare with

The benchmark field starts at 68, which is the figure usually quoted as an average SUS score. It is a reasonable place to start and a poor place to stop, because the population behind an industry average is a set of studies that has nothing to do with your product, your users or the tasks you asked them to do.

The better line to draw is your own score from last time, measured the same way, on the same tasks, with the same recruiting. Then the question stops being whether you beat an average and becomes whether you moved, which is the question anybody was asking in the first place.

What this does not model

Sampling error, and only sampling error.

Who answered is usually the larger problem. A score computed from the people who finished the task describes people who finish tasks.

The tasks move the score more than the sample size does. SUS asked after a guided walkthrough and SUS asked after a cold start are two different measurements wearing the same name, and the interval says nothing about the gap between them.

And an interval that excludes a benchmark says the difference is unlikely to be chance. It says nothing at all about whether the difference is worth anything.