Skip navigation
Nico Santangelo
Get in touch
Instruments

Is this study worth running?

What a study costs against what a wrong build costs, and the reduction in risk it has to deliver to cover itself. Written to be argued with in a budget conversation.

What the study costs
What the decision is worth
What the study changes

The study is worth

€12,660

  • The study costs€6,240
  • A wrong build costs€126,000
  • Return on the study203%

An expected value is an average over outcomes that do not happen. This decision happens once, and the study either changed it or did not.

It pays for itself. To cover its cost it has to cut the chance of a wrong call by 5.0% and you have entered 15.0%, so the case rests on whether 15.0% is a cut you would defend.

Expected saving
€18,900
Smallest decision it pays on
€41,600
Reduction it has to deliver
5.0%

That last figure is the one to take into the room. It is the reduction in the chance of a wrong call at which this study covers its own cost, and unlike a return percentage it is a claim somebody can disagree with from experience.

The same decision under other guesses

Reduction it deliversExpected savingValue of the study
5%€6,300€60
10%€12,600€6,360
15%€18,900€12,660
20%€25,200€18,960
30%€37,800€31,560

The row where the value turns positive is the claim this study has to make about itself. If you would not defend that claim, the model has answered, and it answered while the budget conversation was still open.

What the study is worth, by the size of the decision, in thousands
What the study is worth, by the size of the decision, in thousandsA straight line rising with the size of the decision and crossing zero once, with the decision on the table marked on it.-100102030105294136178210Break-evenYour decision · €12,660

Cost of a wrong build, in thousands

Summary to paste into a ticket

The study is worth: €12,660 (203%)
Reduction it has to deliver: 5.0%
Smallest decision it pays on: €41,600

https://youex.de/tools/research-roi/

The ledger has two sides

Research gets defended with its cost and attacked with its cost, and the other column never gets written down. The model here writes it. A study is worth the chance it removes of building the wrong thing, multiplied by what building the wrong thing would have cost, less what the study costs to run.

Two of those three are ordinary accounting. The study costs participants plus time, and the page counts everybody’s time, the researcher’s and everyone else’s, because the observers and the readout are real money that never appears in a research budget. The wrong build costs its weeks times its team, less whatever survives being wrong, and that last subtraction is there because a wrong build is rarely a total write-off and a model that pretends otherwise overstates the case by a third.

The third number is a guess, and it is the whole argument.

The number nobody can measure

How much does a study lower the chance of a bad call? Nobody knows. There is no counterfactual: the decision was made once, with the research or without it, and the other branch was never run.

So the page works it backwards. Given what the study costs and what the decision is worth, there is a reduction in the chance of a wrong call at which the study exactly covers itself. In the worked example that figure is about five points: the study has to take a forty percent chance of being wrong down to thirty-five.

That is the number to put in front of whoever controls the budget, because unlike a return percentage it is a claim somebody can disagree with from experience. Five points is a modest claim for eight sessions before a ten-week build. Thirty points would not be. If the figure comes out higher than you would defend, the model has answered, and it answered while the budget conversation was still open.

The table underneath is the same decision under other guesses, so you can see how much the answer rests on yours.

Why the chart crosses zero

Plotted against the size of the decision, the value of a fixed study is a straight line: it rises with what is at stake and it starts below zero, because the study costs what it costs however small the decision is.

Where it crosses is the smallest decision this study is worth running on. In the example that is about forty-two thousand euros, and the practical reading is the useful one. Research on a two-week change with a reusable outcome is hard to justify on these numbers. The same study before a ten-week build on a team of six is not close.

That is a more honest conclusion than “research always pays”, and it is the one that survives contact with a finance partner.

What this leaves out, in both directions

Three things make the case for research stronger than the model says.

Opportunity cost is not counted. A team that spends ten weeks on the wrong thing did not spend those ten weeks on the right thing, and that second loss can be larger than the first.

Studies inform more than one decision. The model prices a study against the decision in front of it, and a good interview round changes three roadmap items and a positioning argument.

And the compounding is missing. A team that has talked to users every week for a year makes better calls on decisions this model never sees.

Two things cut the other way.

An expected value is an average over outcomes that do not happen. This decision happens once, and the study either changed it or did not. A portfolio of decisions converges on the average; a single quarter does not have to.

And the study only pays if the decision was open. Research commissioned to confirm a decision already taken has a reduction of zero whatever it finds, and no arithmetic on this page can tell you which kind you are about to run.