The natural logarithm

Before thisread these first

Growth by a constant factorA logarithm is the count of factors that rebuild a number, so it needs powers, the addition law, and where the constant e comes from.
Reading a formulaThe penalty over a sequence is written as a sum with a counter, and that shorthand is built there.

After thiswhat you will be able to doRun a power backward with a logarithm, turn a product of shares into a sum of penalties, and explain why the sum avoids machine underflow.

Questionwhat this lesson answersGrowing by a constant factor turns a count into a size. What runs that backwards, and why does running it backwards turn multiplication into addition?

Not coveredwhat this lesson leaves outWe build the count a logarithm reports, the product rule, and the penalty read of a negative logarithm. We do not use calculus, so we do not show why the natural logarithm is the one with the tidiest slope. We do not average a penalty over a body of text. We do not derive how a machine stores a number, only what happens when one gets too small to store.

You know that something doubles each step, and you know it ended up 1000 times bigger. How many steps was that?

Powers answer the other question. Give them the count and they give you the size. This one runs the other way: you have the size and you need the count.

A logarithm is a count of factors

The logarithm of a number is the count of equal factors that rebuild it.

Three twos multiplied together make 8, so the logarithm of 8 in base 2 is 3. Written down:

2^3 = 8 \qquad\text{means}\qquad \log_2 8 = 3.

They are the same fact, read in opposite directions. The lesson on logarithms counts halvings this way and shows that changing the base only multiplies every count by a fixed number, so nothing about the shape changes.

When the base is ${e}$ , the count is called the natural logarithm and written ${\ln}$ . That is the only difference. It counts factors of ${e}$ instead of factors of 2.

Multiplying inside becomes adding outside

Take two numbers, each built from factors of the same base. Multiply them.

Say the first is 4 factors and the second is 3 factors. Multiplying them puts all 7 factors in one pile. So the count for the product is the sum of the two counts:

\ln(ab) = \ln a + \ln b.

Nothing was assumed to get this. It follows from what a power is, which the lesson on growth by a constant factor established: joining ${a}$ factors to ${b}$ more factors gives ${a+b}$ factors. The logarithm reports counts, and counts add.

That one line is why logarithms are everywhere. Any time you have a long product and would rather have a long sum, this is the move.

Now put the logarithm to work on a share.

A share sits between 0 and 1. Its logarithm is therefore negative, since you need a negative count of factors to get below 1. Flip the sign and you have a positive number:

\text{penalty} = -\ln p.

Look at what that number does. A share of 1, meaning the answer was certain and it was right, gives a penalty of exactly 0. A share of ${0.5}$ costs about ${0.69}$ . A share of ${0.01}$ costs about ${4.61}$ . As the share falls toward zero the penalty climbs with no ceiling at all.

What a share costs

A right answer held with certainty costs nothing. Everything else costs something, and the cost has no upper limit. The curve is drawn from a share of 0.004 rather than from zero, because at zero it has no height to draw.

Why nobody multiplies these

The same share, 2E-5, multiplied by itself once per step, against the same information added up as penalties.

steps	1
shares multiplied	2E-5
penalties added	10.82

Both still carry the answer.

That shape was chosen, not discovered. Somebody wanted a score that is nothing when you were certain and right, that grows as you were less sure, and that has no upper limit for being confidently wrong. The negative logarithm has all three properties. Other scores are possible, and whether this one produces a system anyone wants is a separate question answered by measurement.

Why a sum instead of a product

A sequence of text is thousands of tokens long, and each one carries a share. To score the whole sequence you want all those shares combined, which means multiplying them together.

That does not work on a machine. A machine holds a number in a fixed amount of space, so once a number gets small enough it stops being a small number and becomes exactly zero. Multiply a share of one in fifty thousand by itself a hundred and forty times or so and the running product hits zero. Everything after that point is lost: a very unlikely sequence and a completely impossible one now score identically.

The island’s second panel runs exactly that. Watch the product collapse while the added penalties carry on counting.

Adding the penalties instead sidesteps the whole problem, because a sum of ordinary-sized numbers stays an ordinary-sized number. It is not an approximation. By the product rule, the sum of the penalties is the penalty of the product, so nothing has been given up.

What this does not settle

It does not say why ${e}$ is the natural base. That answer is about slopes and needs calculus.

It does not say what a machine can hold or why a number stops fitting, only that it happens and what it costs.

And it does not make the penalty meaningful. The score is well defined for any set of shares, including shares that have nothing to do with how often anything really happens.

Doorswhat to read next, and why

Logarithms, and why the base stops matteringThat lesson counts halvings in base two and shows that changing base only multiplies the count by a fixed number. This one takes the base as e and does not rebuild that argument.
Pretrainingnot written yetThis lesson builds logarithmic penalties, but it does not explain how a pretraining objective applies a loss across a large dataset or what that process costs at scale.
How a machine stores a numberThis lesson shows a product reaching exactly zero and takes that behaviour as given. What a machine can and cannot hold is not built here.
A distribution fixes the totalThe penalty is applied to shares that add to one, and what a share is and does not promise is built there.
A slope is a local statementThe reason this base rather than another is the natural one is a fact about slopes, and it needs calculus this lesson does not use.

Symbolswhat each one means, and whether we defined it, measured it, or just started there

log of xStatus: defined: The count of equal factors that rebuild x. Asking for the logarithm is asking what power the base was raised to, so it is defined as the reverse of a power and nothing more.
the natural logarithmStatus: defined: The logarithm whose base is e, written ln. Defined by the choice of base, and that choice changes every count by one fixed multiplier rather than changing what is being counted.
log of a times b equals log a plus log bStatus: defined: This follows from the addition law for powers rather than being a separate assumption. Joining a factors to b more factors gives a plus b factors, and the logarithm reports that count.
the penalty, minus log of a shareStatus: defined: A way of scoring an answer, chosen because it is zero at certainty and has no ceiling. It is a choice about how to score, not something the mathematics forced.
a product of many shares reaching exactly zeroStatus: empirical: A machine holds numbers in finite space, so a small enough number becomes zero rather than becoming smaller. That is a measured fact about real machines and not a truth about arithmetic.
whether this penalty is the right score for a taskStatus: empirical: Whether scoring answers this way produces a system anyone wants is a question about the task, answered by measurement rather than by the definition.

What these classifications mean

defined: circular by construction, true because we chose it
empirical: a measured claim about the world that could have come out otherwise
bottoms out: a primitive of the model, with nothing under it here
door: used here, explained elsewhere