Two rules fix the whole transform

After thiswhat you will be able to doTest whether a transform preserves sums and scaling, reconstruct its matrix from simple inputs, and explain why stacking such transforms cannot bend a line.

Questionwhat this lesson answersAmong all the ways to turn one list of numbers into another, what do preserving sums and preserving scale force a transform to do, and what remains impossible after any number of them are applied in a row?

Not coveredwhat this lesson leaves outWe carry out the mathematics only with lists of two or three numbers, called two and three dimensions, and give no general treatment of dimension. We do not derive the general rule for combining two tables. We name but do not develop the single number that detects flattening, called the determinant, the special directions called eigenvectors, or matrices that undo other matrices, called matrix inverses. We also do not prove the general results about what a flattened transform loses.

A point on a map needs two coordinates at once. A recipe may need the amounts of three ingredients at once. Four sensor readings may describe one machine at one moment. In each case, separating the numbers would throw away the fact that they belong to the same thing.

Treat the whole ordered list as one object. That object is called a vector. The number of entries it holds is called its dimension. We will do the arithmetic with pairs of numbers and occasionally picture triples, but the reason for grouping them does not depend on the example.

Now suppose you want to turn one such object into another. Coordinates might need turning, an ingredient mixture might need rescaling, or sensor readings might need combining. Most possible transforms have no useful common structure. One family is picked out by two rules.

Choose two rules

Add vectors by adding matching entries. Multiply a vector by one number by multiplying every entry. A transform $T$ belongs to the family only when

T(u+v)=T(u)+T(v)

and

T(\lambda v)=\lambda T(v).

The first rule says that transforming a sum gives the same answer as transforming the parts and adding them. The second says that scaling the input scales the output by the same amount. These are chosen conditions, not discoveries. A transform obeying both is called a linear map.

The rules exclude familiar operations. Let $Q$ square the first entry and leave the second alone. It fails the sum rule:

\begin{aligned} Q((1,0)+(1,0)) &= Q(2,0) = (4,0),\\ Q(1,0)+Q(1,0) &= (1,0)+(1,0) = (2,0),\\ Q(2(1,0)) &= (4,0),\\ 2Q(1,0) &= (2,0). \end{aligned}

Both pairs of routes disagree, so $Q$ fails both rules and is not a linear map. Whether a relationship measured in the world obeys the rules is likewise not settled by vocabulary. It is an empirical question.

The rules leave little freedom

Start with the vector whose entries are all zero. This point is called the origin. The sum rule forces

T(0)=T(0+0)=T(0)+T(0).

Subtract ${T(0)}$ from both sides and only ${T(0)=0}$ remains. The origin cannot move.

In two dimensions, take the two simplest arrows: ${(1,0)}$ and ${(0,1)}$ . Call them $e_1$ and $e_2$ . Every other input is built from scaled copies of them:

(x,y)=x(1,0)+y(0,1).

Decide that the first arrow lands at ${(2,0)}$ and the second at ${(1,1)}$ . The two rules then fix every output:

\begin{aligned} T(x,y) &= xT(1,0)+yT(0,1)\\ &= x(2,0)+y(1,1)\\ &= (2x+y,y). \end{aligned}

For example, ${T(3,2)=(8,2)}$ . There is nothing left to choose.

The four numbers can be recorded in a small table, with each landing place written as one column:

A= \begin{pmatrix} 2 & 1\\ 0 & 1 \end{pmatrix}.

Such a table is called a matrix. It is not a different kind of transform. It is a compact record of what the transform does to the simple arrows.

A straight line is made by starting somewhere and adding scaled copies of one direction. A linear map sends the start somewhere, sends the direction somewhere, and leaves every scale factor in place. The result is still a straight line, unless it has been flattened to one point. Equal steps along the input line therefore remain equal steps along the output line.

What the table can and cannot do

By choosing its two columns, a matrix can stretch, squash, turn, or reflect a shape. It can also slide each horizontal level sideways by an amount set by its height, an operation called a shear, or flatten the whole shape. A quarter turn has the table

R= \begin{pmatrix} 0 & -1\\ 1 & 0 \end{pmatrix}, \qquad R(x,y)=(-y,x).

So ${(2,1)}$ lands at ${(-1,2)}$ . Flattening everything onto the horizontal line uses

P= \begin{pmatrix} 1 & 0\\ 0 & 0 \end{pmatrix}, \qquad P(x,y)=(x,0).

The same rules forbid three things. The origin cannot move. A straight line cannot bend. A straight relationship between inputs cannot become a curved one. No clever choice of the four numbers changes those consequences.

Flattening cannot be undone

The table $P$ sends the vertical direction ${(0,1)}$ to zero. Moving any distance vertically before the transform changes nothing afterwards. Many different inputs now give the same output, so the input cannot be recovered. The transform has no undo.

For a two-dimensional table with rows ${(a,b)}$ and ${(c,d)}$ , multiply the top-left entry by the bottom-right entry, then subtract the top-right entry times the bottom-left: ${ad-bc}$ . This number is called the determinant. When it is zero, the two simple arrows land along one line and the table flattens. This lesson uses that test without developing determinant arithmetic or a general proof of what has been lost.

Two maps still make one map

Applying one transform and then another is called composition. Apply a linear map $A$ , then another linear map $B$ , and call the complete operation $C$ . The sum rule survives the sequence:

\begin{aligned} C(u+v) &= B(A(u+v))\\ &= B(A(u)+A(v))\\ &= B(A(u))+B(A(v))\\ &= C(u)+C(v). \end{aligned}

The scaling rule survives by the same route:

\begin{aligned} C(\lambda v) &= B(A(\lambda v))\\ &= B(\lambda A(v))\\ &= \lambda B(A(v))\\ &= \lambda C(v). \end{aligned}

So $C$ is itself a linear map. Its matrix can be worked out from the other two. Use the earlier matrix for $A$ and the quarter turn for $B$ :

\begin{aligned} A(x,y) &= (2x+y,y),\\ B(A(x,y)) &= (-y,2x+y),\\ C &= \begin{pmatrix} 0 & -1\\ 2 & 1 \end{pmatrix}. \end{aligned}

Move two arrows, fix the whole map

One table, then two, then one again

The solid accent endpoint is where the input (1, 0) lands. The dashed endpoint is where (0, 1) lands. Drag either endpoint, or use the four sliders. Every line in the square follows because every point in it is built by scaling and adding those two inputs.

The two landing places become the columns of matrix A. Choose a preset, drag a point, or change one coordinate with a slider. Each slider accepts arrow keys.

First map, A

(1, 0) horizontal2

(1, 0) vertical0

(0, 1) horizontal1

(0, 1) vertical1

Not flattening

The determinant is 2, so no non-zero direction is sent to zero.

Second map, B

Input grid

After A

After A, then B

One combined matrix C

First table A

2101

Second table B

0−110

Combined table C

0−121

Across every endpoint in the drawn grid, the largest difference between applying A then B and applying C once is 0. The dashed two-step grid and the solid single-table grid coincide.

This is the permanent limit. Stacking these transforms buys nothing that one of them could not already do. A hundred linear maps in a row are still one linear map. The stack still cannot move the origin or bend a line. To produce a bend, it needs an operation outside this family between the maps, such as squaring an entry or setting every negative entry to zero.

The common near miss

One small change needs separating cleanly. Apply a linear map and then add the same fixed, non-zero vector $h$ to every output. The origin now lands at $h$ , not at zero. The sum rule fails because

\begin{aligned} F(u+v) &= A(u)+A(v)+h,\\ F(u)+F(v) &= A(u)+A(v)+2h. \end{aligned}

This combination is called an affine map. It is extremely common. A straight line may cross the vertical axis away from zero. That crossing value is called its intercept. What people call a linear trend in ordinary speech is usually affine: a line with a non-zero intercept fails the two rules used here unless it passes through the origin.

Doorswhat to read next, and why

The transformer blocknot written yetThis lesson builds linear maps in isolation, but it does not explain how a transformer block combines learned maps with attention, a feedforward network, residual connections, and layer normalisation.
Attentionnot written yetThis lesson builds learned maps, but it does not explain how attention uses separate query, key and value maps to connect one token with others.
Turning words and images into numbersThis lesson treats vectors abstractly, but it does not explain how words and images are encoded as vectors or why similar words can end up near one another.
Derivatives and slopesA curved relationship can be replaced near one point by its best straight-line stand-in, which is how maps of this kind remain useful where they are not exact.
Eigenvectors and determinantsThe special directions that a transform only stretches are called eigenvectors, and the single number that detects flattening is the determinant. Both are named here at most and neither is developed.

Symbolswhat each one means, and whether we defined it, measured it, or just started there

v = (x, y)Status: defined: A vector v is defined here as one ordered group of numbers treated as a single object, with x and y naming its two entries.
uStatus: defined: The symbol u is defined as a second vector so that the rule about adding two inputs can be stated.
lambdaStatus: defined: The number lambda is defined as the single scale factor multiplying every entry of a vector.
TStatus: defined: The symbol T is defined as the transform being tested against the two chosen rules.
QStatus: defined: The transform Q is defined as the counterexample that squares its first input entry and leaves its second entry alone.
T(u + v) = T(u) + T(v)Status: defined: Preserving sums is one of the two conditions chosen to define a linear map, so it is true of this family by definition.
T(lambda v) = lambda T(v)Status: defined: Preserving scaling is the other condition chosen to define a linear map, so it is true of this family by definition.
0Status: defined: The symbol 0 is defined as the vector whose entries are all zero, which is the point called the origin.
T(0) = 0Status: defined: That the origin cannot move follows from the two defining rules, so it is a consequence of definitions rather than a measured claim.
e_1 and e_2Status: defined: The vectors e_1 and e_2 are defined as the simple two-dimensional inputs (1, 0) and (0, 1).
A = [[a, b], [c, d]]Status: defined: The matrix A is defined as the two-by-two table whose columns record where the two simple inputs land.
RStatus: defined: The matrix R is defined as the table that turns every two-dimensional input through one quarter turn.
PStatus: defined: The matrix P is defined as the table that keeps the horizontal entry and sends every vertical entry to zero.
B(A(v)) = C(v)Status: defined: Applying one map after another is called composition. It stays inside the family because both maps obey the same two rules, and C is defined as the resulting single map.
ad - bc = 0Status: defined: The number ad minus bc is defined as the determinant of the two-dimensional matrix, and zero is the condition used here for flattening.
a direction sent to zeroStatus: defined: A flattening transform is defined to lose a direction when every scaled copy of that non-zero direction produces the zero output, so inputs separated along it cannot be recovered.
F(v) = A(v) + hStatus: defined: An affine map F is defined here as a linear map A followed by adding one fixed offset vector h.
whether a measured relationship obeys both rulesStatus: empirical: Whether quantities in the world actually preserve sums and scaling must be checked against observations and can turn out to be false.

What these classifications mean

defined: circular by construction, true because we chose it
empirical: a measured claim about the world that could have come out otherwise
bottoms out: a primitive of the model, with nothing under it here
door: used here, explained elsewhere