Maths/Distributions

Telling a real difference from noise

After thiswhat you will be able to doCompare two finite-trial success counts, read how often chance makes the worse option win, and estimate how trial count changes when the true rate gap is halved.

Questionwhat this lesson answersTwo options are tried the same number of times and one records more successes. When does that mean it is better, and how many tries does it take before the answer is worth anything?

Not coveredwhat this lesson leaves outWe compare two rates over a fixed number of tries and count how often the comparison points the wrong way. We do not build the machinery for testing a claim formally, which has its own vocabulary. We do not cover stopping early when a result looks good, which breaks everything here. We do not handle more than two options at once.

Two ways of doing something. You try each one ten times. The first succeeds 6 times, the second 4 times.

The first one looks better. Is it?

A measured rate is not the true rate

Every option has some true rate: how often it would succeed over an unlimited number of tries. That number is a fact about the option and you never get to see it.

What you see is a measured rate, which is successes divided by tries in the run you actually did. Do the run again and you get a different number.

So the question is not whether 6 out of 10 beats 4 out of 10. It plainly does. The question is whether that gap tells you anything about the two true rates underneath.

Watch the two answers overlap

Take an option with a true rate of ${0.52}$ and one with a true rate of ${0.50}$. The first really is better. Give each of them 10 tries and count.

The two curves are what the measured rates do. At ten tries they sit almost on top of each other, and that overlap is the whole problem. Both options can produce almost any count, so the count you happen to see says very little about which curve it came from.

Counting every possible pair of results gives three outcomes:

Read those again. The genuinely worse option comes out ahead more than a third of the time. And almost a fifth of the time the comparison picks nobody at all.

Throw away the ties and look only at the runs that chose a winner. The worse option is the winner ${45.7301\%}$ of the time. That is very nearly a coin toss, from a test that felt like evidence.

Buying certainty with tries

The curves pull apart as the number of tries grows. Watch the island step up through the counts: the two humps separate and each one narrows.

So the fix is more tries. The question is how many.

Ask for the better option to win outright at least ${95\%}$ of the time. With a gap of two percentage points, that takes 3,430 tries each.

Now ask to catch a gap of one percentage point instead, at the same confidence. That takes 13,626 tries each.

Halving the difference did not double the work. It multiplied it by about four.

That is the shape of the whole business. The tries needed grow with the square of how small a difference you want to catch. Chasing a difference ten times smaller costs a hundred times as much work.

What this does not give you

It does not tell you whether the difference you found is real. Every number here assumes you already know the two true rates and asks how often the comparison gets it right. In practice you have one run and no idea what is underneath it.

It also assumes you fixed the number of tries before you started. Checking as you go and stopping the moment the answer looks good breaks every figure on this page, because you get many chances to catch a favourable wobble instead of one. That repair has its own name, sequential testing, and it is not built here.