How a machine stores a number

After thiswhat you will be able to doExplain why fixed binary storage cannot represent every number, and predict when rounding makes regrouped sums disagree or small values become zero.

Questionwhat this lesson answersA machine keeps every number in a fixed amount of space. What can fit in that space, and what happens to the arithmetic when a number does not?

Not coveredwhat this lesson leaves outWe build the bit, the byte, why a tenth cannot be stored, and the three ways stored arithmetic disagrees with arithmetic on paper. We do not lay out the exact bit pattern of a stored number. We do not cover the special values a machine keeps for infinity and for results that are not numbers. We do not give methods for arranging a calculation to lose less.

Ask any computer to add a tenth to two tenths. It will tell you the answer is

0.30000000000000004.

Not a rounding error in the display. That is the answer it holds. And it is not a bug in any particular machine: nearly every computer you will ever use does this, and does it for a reason worth knowing.

Everything is counted in two-way choices

The smallest thing a machine stores is a single two-way choice: on or off. That is a bit, and nothing here is built from anything smaller.

One bit holds 2 possibilities. Add a second and everything that could already be written can be followed by either an on or an off, so the count doubles to 4. A third doubles it again to 8.

Eight bits together are called a byte. Doubling once per bit gives

2^8 = 256

different values in one byte. That is where 256 comes from every time it turns up, including in the vocabulary lesson, which starts from one entry per possible byte value.

Eight is a convention. Hardware settled on it, and nothing about numbers required it.

A fixed space means a fixed grid

Here is the part that causes the trouble.

A machine gives each number a fixed amount of space. Fixed space means a fixed count of possible values, so the numbers a machine can hold are not all the numbers. They are a grid of particular values with gaps between them.

Write a number that lands on the grid and it is stored exactly. Write one that falls in a gap and it cannot be stored at all, so it moves to the nearest grid point instead. That move is called rounding, and it happens before your calculation has even started.

Where a written number has to land

The marks are the values that can be held exactly. They are drawn far apart here so they can be seen. The real grid is very much finer, and the argument does not change: a number written between two marks has to move to one of them.

What actually got stored

written	0.5
stored	0.50000000000000000000
exact	yes

This one has a denominator that is a power of two, so base two writes it exactly.

Which numbers land on the grid? Base two writes exactly the fractions whose denominators are powers of two. A half, a quarter, an eighth: all exact. A tenth is not, because however far you reduce it, it keeps a factor of five.

This is not exotic. Base ten has exactly the same problem with a third, which is why ${1/3}$ has no exact decimal and runs ${0.333\ldots}$ forever. Base two simply has the problem with a different set of fractions, and a tenth is unluckily one of them.

So ${0.1}$ was never in the machine. What went in was the nearest grid point, which is a hair above a tenth. Add it to a similarly inexact two tenths and the result misses three tenths.

Three consequences

Regrouping a sum can change the answer. On paper, ${(a + b) + c}$ and ${a + (b + c)}$ are always equal. Stored, each addition rounds its result, and the two routes round at different moments, so they can disagree. Most of the time they agree, which is what makes the exceptions dangerous: nobody checks.

Whole numbers stop being exact past a certain size. Below about nine thousand million million, every whole number sits on the grid. Above it the gaps grow wider than one, so two different whole numbers land on the same stored value, and adding one to a large enough number changes nothing at all.

Small numbers stop being small and become zero. The grid does not extend forever downward either. Multiply enough shares together and the result eventually reaches exactly zero rather than becoming a smaller number. Everything after that is lost, which is why the natural logarithm lesson adds penalties instead of multiplying shares.

Sizes, and why they are not round

Storage gets counted in these units too, so the numbers look odd until you know why.

A kibibyte is ${2^{10} = 1024}$ bytes. A mebibyte is ${2^{20} = 1{,}048{,}576}$ bytes. Neither is a round decimal, because neither was ever chosen in base ten. They are the powers of two closest to a thousand and a million, and they got their names afterwards.

What this is not

None of this is a defect being tolerated. It is the price of holding a number in a fixed amount of space, and every alternative costs something else: more space per number, or slower arithmetic, or a limit on which numbers can appear at all.

The rules themselves are a written standard that hardware and languages agree to follow. Whether a particular machine or a particular setting actually follows it is a question about that machine, and the answer has occasionally been no.

Doorswhat to read next, and why

Growth by a constant factorThe count of values a run of bits can hold doubles with every extra bit, and repeated doubling is built there.
The natural logarithmThat lesson multiplies many small shares together and watches the product reach exactly zero. This lesson says why a machine does that.
The context problemnot written yetThis lesson explains finite storage for one number, but it does not explain the separate limit on how much text a model can read in one pass.
Tokenizationnot written yetThis lesson builds bytes, but it does not explain how byte pair encoding groups text into subword tokens or what changing the vocabulary size costs.
Arranging a calculation to lose lessSome orders of adding the same numbers lose more than others, and choosing a good order is a craft this lesson does not teach.

Symbolswhat each one means, and whether we defined it, measured it, or just started there

a bitStatus: bottoms out: One two-way choice, on or off. Nothing here is built out of anything smaller, so this is where the account starts.
a byteStatus: defined: Eight bits treated as one unit. Eight is a convention that hardware settled on rather than a fact about numbers, and everything about it follows from that choice.
256 values in a byteStatus: defined: This follows from eight bits by doubling once per bit. It is a consequence of the convention, not a separate fact.
a tenth having no exact stored valueStatus: defined: Base two writes exactly those fractions whose denominators are powers of two. A tenth keeps a factor of five, so it cannot be written, in the same way base ten cannot write a third.
rounding to the nearest storable valueStatus: defined: A number that falls between two storable values is moved to one of them. Defined by the storage rules, and the amount moved is decided by the grid rather than by the calculation.
regrouping a sum changing the answerStatus: defined: On paper the grouping of a sum never matters. Stored, each addition rounds, so two groupings can round at different moments and disagree. It follows from rounding rather than being a separate defect.
whether a particular machine behaves this wayStatus: empirical: These rules are a written standard that hardware and languages agree to follow. Whether any given machine or setting follows it is a fact about that machine, which can be checked and has sometimes been false.

What these classifications mean

defined: circular by construction, true because we chose it
empirical: a measured claim about the world that could have come out otherwise
bottoms out: a primitive of the model, with nothing under it here
door: used here, explained elsewhere