Maths/Vectors and linear maps

Comparing whole lists of numbers

After thiswhat you will be able to doCalculate the length, dot product, and cosine similarity of two vectors, and explain why cosine similarity ignores their size but cannot validate what their entries mean.

Questionwhat this lesson answersWhen a list becomes one object, how can its arithmetic compare two patterns, and what can that comparison never tell you?

Not coveredwhat this lesson leaves outWe do not prove the general bound that keeps cosine similarity between -1 and 1. We do not derive cosine from circle geometry. We do not decide where useful vector entries should come from.

Imagine two tiny black-and-white pictures, each made from three pixels. Larger numbers mean brighter pixels. The first picture gives 1, 2, 2. The second gives 2, 1, 2.

How alike are the pictures? No single position answers that question. The first positions differ, the second positions differ, and the third positions match. We need arithmetic that keeps those three comparisons together.

The whole ordered list becomes one vector

Treat the whole ordered list as one object. That object is called a vector. The number of entries it holds is called its dimension. Each picture here gives a vector of dimension three.

Order matters. The first entry of one vector matches the first entry of the other. Rearranging one list would change which pixels meet.

Adding and scaling work entry by entry

Add the two picture vectors by adding matching entries:

Multiply one vector by 2 by multiplying every entry by 2:

This second operation is called scaling. One number stretches every entry together.

In two dimensions, ${(2,1)}$ can be drawn as an arrow two steps across and one step up. To add ${(1,2)}$, place its tail at the first arrow’s head. The arrow from the original start to the new finish is ${(3,3)}$. Scaling ${(2,1)}$ by 2 gives ${(4,2)}$, which points the same way and reaches twice as far.

Past three dimensions, this arrow picture stops being drawable. The arithmetic does not stop working. A vector with 768 entries still adds and scales by the same entry-by-entry rules.

Length extends the Pythagorean recipe

Draw the two-dimensional arrow ${(3,4)}$. It forms a right-angled triangle with horizontal side 3 and vertical side 4. The theorem of Pythagoras says its squared length is the sum of the squared sides:

The raised 2 means multiply a number by itself. The square root of 25 is the non-negative number which, multiplied by itself, gives 25. Here that number is 5.

This length is a theorem about distance in the picture. Apply the same recipe to the three-dimensional vector ${(1,2,2)}$:

For more entries, the recipe stays unchanged. Square every entry, add the squares, then take the square root. Beyond drawable dimensions, this is a definition chosen because it agrees with visible distance where the picture exists.

The dot product multiplies matching entries

Return to the picture vectors. Multiply entries in matching positions, then add those products:

This multiply-and-add recipe is called the dot product. One number comes out.

It is large and positive when large entries occur in the same places. It is zero when the lists have no overlap in that sense:

It is negative when the lists run in opposite directions:

A raw dot product also measures size

The original dot product is 8. Double every entry in the first vector and repeat the same work:

The pattern did not change. Every entry kept the same proportion to the others. Yet the dot product doubled. A raw dot product mixes up how alike two patterns are with how big they are.

Dividing by both lengths removes size

The two original vectors both have length 3. The doubled first vector has length 6. Divide each dot product by the two lengths involved:

Both results are about 0.89. Doubling changed the dot product from 8 to 16, but the divided result did not move. Any positive scaling of either vector cancels in the same way.

For any two non-zero vectors, the result lies between -1 and 1. The earlier zero-overlap pair gives 0 after division. The opposite pair gives -1. Comparing any non-zero vector with itself gives 1.

This size-free result is called cosine similarity. In two dimensions, it matches the cosine of the angle between the arrows. The word cosine is borrowed from that picture. The circle arithmetic behind the name is a separate lesson.

If every entry is zero, the length is zero. Division by that length is impossible, so cosine similarity is undefined for the zero vector.

Now give the compact version names. Let ${a}$ and ${b}$ name the two vectors. Their entries at position ${i}$ are ${a_i}$ and ${b_i}$. The summation sign ${\sum_i}$ adds the result from every position. Writing ${a_i b_i}$ means multiply those matching entries. The notation ${\text{similarity}(a,b)}$ names the score produced from both lists.

The expression above the fraction bar is the dot product built by matching, multiplying, and adding. Each square-root expression is one vector length. Their product is the size being divided out. The fraction bar divides the dot product by that product of lengths.

A score cannot explain where its entries came from

Two lists can point in exactly the same direction and mean completely different things. The same three entries could record pixels or survey answers. The arithmetic cannot see those labels.

The numbers had to come from somewhere, and this lesson takes them as given. Whether the similarity tracks something a person cares about must be checked against observations. That claim is empirical and could prove false. It depends entirely on how those entries were produced.

A similarity of 0.9 is not a probability. It is not 90 percent agreement. It reports only how two non-zero vectors line up under this chosen arithmetic.