Average and spread

Before thisread these first

Reading a formulaThe mean and the variance are both written as a sum over a list with a counter, and that shorthand is built there.
Drawing at randomThe result at the end of this lesson only holds for independent quantities, and independence is defined there.
A distribution fixes the totalA list of shares over outcomes is one of the things whose average and spread this lesson measures.

After thiswhat you will be able to doCalculate the mean, variance, and standard deviation of a finite list, and use added variances to find the spread of a sum of independent quantities.

Questionwhat this lesson answersOne number can stand in for a whole list, and a second number can say how badly it stands in. What are those two numbers, and what does each one hide?

Not coveredwhat this lesson leaves outWe build the mean, the deviations, the variance and the standard deviation for a list we already hold. We do not prove that variances add, we demonstrate it. We do not estimate the spread of a larger group from a sample, which needs a different divisor and a different lesson. We do not describe the shape of a list beyond these two numbers.

Two lists, five numbers each.

19,\ 20,\ 20,\ 21,\ 20 \qquad\text{and}\qquad 4,\ 33,\ 12,\ 28,\ 23.

Both add to 100. Both have five entries. So both have the same average, 20.

Nobody would call these lists alike. The average is 20 for both, so it cannot separate them. A second number is needed.

The mean

Add the values, divide by how many there are. For the first list,

\dfrac{19 + 20 + 20 + 21 + 20}{5} = \dfrac{100}{5} = 20.

That is the mean, also called the average. In shorthand, for a list of ${n}$ values,

\text{mean} = \dfrac{1}{n}\sum_{i=1}^{n} x_i.

The mean is a summary, so it leaves things out. Both lists above summarise to 20. The question is what to measure next.

Distances from the mean cancel

One way to measure spread is to take how far each value sits from the mean, then average those distances.

Take the scattered list. Its mean is 20, so the distances are

-16,\quad 13,\quad -8,\quad 8,\quad 3.

Add them: ${-16 + 13 - 8 + 8 + 3 = 0}$ .

Try the tight list and the same thing happens. The distances are ${-1, 0, 0, 1, 0}$ and they add to zero too.

It happens every time, and it happens because of what the mean is. The mean sits exactly where the pull from below matches the pull from above. So the average distance from the mean is always zero, for every list, and it carries no information at all.

The signs are the problem. Something has to remove them.

Squaring, and why it is a choice

There are two obvious ways to remove a sign. Take each distance’s size and ignore whether it was negative. Or square it, since a negative number times itself is positive.

Both are used in practice. This lesson squares, and the reason is narrow: the result at the end of this lesson is true for squares and false for sizes. Squaring is chosen because of where it leads, not because it is more natural.

Square the scattered list’s distances:

256,\quad 169,\quad 64,\quad 64,\quad 9.

Average them: ${(256 + 169 + 64 + 64 + 9)/5 = 562/5 = 112.4}$ . That number is the variance.

Two lists the mean cannot tell apart

tight: mean 20.0, spread 0.63

scattered: mean 20.0, spread 10.60

List	Values	Mean	Variance	Spread
tight	19, 20, 20, 21, 20	20.0	0.4	0.63
scattered	4, 33, 12, 28, 23	20.0	112.4	10.60

Same count, same mean of 20.0. The spread separates them: 0.63 against 10.60.

Why the deviations get squared

The scattered list, value by value
Value	Distance from mean	Squared
4.0	-16.0	256.0
33.0	13.0	169.0
12.0	-8.0	64.0
28.0	8.0	64.0
23.0	3.0	9.0
total	0.0	562.0

The distances add to zero every time, so their average says nothing. Squaring removes the signs, and the average of the squared column is the variance.

Adding independent quantities

Quantities combined: 1

Totals: 2, -2, 2, -2, 2, -2, 2, -2

Combined	Actual spread	Root rule	If spreads added
1	2.00	2.00	2.00
2	2.83	2.83	4.00
3	3.46	3.46	6.00
4	4.00	4.00	8.00
5	4.47	4.47	10.00
6	4.90	4.90	12.00

The actual spread tracks the root rule, not the last column. Spreads do not add. Variances do, and the spread is the square root of the variance.

The variance is in the wrong units. The values were measurements; their squares are squared measurements, and 112.4 cannot be compared against a value of 33 in any meaningful way. So take the square root:

\sqrt{112.4} \approx 10.60.

That is the standard deviation, and it is what people usually mean by the spread. Do the same for the tight list and its variance is 0.4, giving a spread of about 0.63.

Two lists with the same mean, and spreads of 10.60 against 0.63. The spread separates them.

Independent quantities have variances that add

Take several quantities that are independent, meaning that knowing how one came out tells you nothing about the others. Add them together. What is the spread of the total?

Spreads do not add. That is the answer most people expect, and it is wrong.

What adds is the variance. Combine four independent quantities each with a spread of 2, so each with a variance of 4. The total has a variance of ${4 \times 4 = 16}$ , so its spread is

\sqrt{16} = 4,

not 8. The island’s third panel shows this happening for one through six quantities. The actual spread tracks the square root, never the sum.

In general, adding ${n}$ independent quantities each of spread ${s}$ gives a total of spread

s\sqrt{n}.

This lesson demonstrates that rather than proving it. The proof needs a way to say how two quantities move together, which is not built here.

The condition is as important as the formula. If the quantities are not independent, this is simply false. Quantities that rise and fall together produce a total that swings far more wildly than the square root suggests, and every serious failure of this result in practice is a case where somebody assumed independence that was not there.

What these two numbers still hide

They do not tell you the shape. Two lists can share a mean and a standard deviation and look nothing alike, and neither number would notice.

Neither one shrugs off an extreme value. Change the scattered list’s 33 to 200 and the mean moves from 20 to 53.4 while the spread moves from 10.60 to about 73.78. One entry rewrote both summaries.

And a spread worked out from five values is itself a rough guess about whatever those five values came from. How rough it is, and how many values you need before it is any good, is the subject of telling a real difference from noise, telling a real difference from noise.

Doorswhat to read next, and why

Attentionnot written yetThis lesson builds ordinary averages, but it does not explain how attention assigns weights to earlier tokens and combines their value vectors for each later position.
The transformer blocknot written yetThis lesson builds the arithmetic of spread, but it does not explain how a transformer applies that arithmetic per position through layer normalisation.
Telling a real difference from noiseA spread worked out from a handful of values is itself a rough guess, and this lesson does not say how rough.
Proof that variances addThis lesson demonstrates the result on worked lists and does not prove it. The proof needs an account of how two quantities move together, which is not built here.

Symbolswhat each one means, and whether we defined it, measured it, or just started there

the meanStatus: defined: The total of the values divided by how many there are. It is defined that way, and no measurement could contradict it.
the deviations adding to zeroStatus: defined: The distances from the mean always cancel. This follows from the definition of the mean rather than being a separate fact, which is why their plain average is useless.
squaring the deviationsStatus: defined: Squaring is a choice, not something forced. Measuring each distance by its size alone would also remove the signs. Squaring is chosen because it is what makes the result at the end of this lesson true.
the varianceStatus: defined: The average of the squared distances from the mean. Defined, and measured in the square of whatever units the values were in.
the standard deviationStatus: defined: The square root of the variance, defined so the spread can be compared against the values themselves rather than against their squares.
variances of independent quantities addingStatus: defined: This is a theorem, provable from the definitions, and no experiment could refute it. This lesson demonstrates it on worked lists instead of proving it, and the proof is named as a door.
whether real quantities are independentStatus: empirical: Whether any particular set of quantities actually satisfies the condition is a claim about the world that can be false, and it is usually the part that turns out to be wrong.

What these classifications mean

defined: circular by construction, true because we chose it
empirical: a measured claim about the world that could have come out otherwise
bottoms out: a primitive of the model, with nothing under it here
door: used here, explained elsewhere