Generative AI, From Scratch/Part 1: Foundations

Turning words and images into numbers

After thiswhat you will be able to doExplain what an embedding is, measure similarity with arithmetic, say where the numbers come from, and explain what dimension means.

Questionwhat this lesson answersHow do words and images become numbers that preserve useful similarity, and what does dimension mean?

Not coveredwhat this lesson leaves outWe do not train anything, build the mechanism that produces the clustering, cover attention, or cover the image work beyond naming it.

Where this lesson is going

Lesson 1 ended with a specific failure. The counting model gave “elephant” a probability of exactly zero after “the”, because it had never seen that pair, and it had no way to notice that elephant behaves like dog. Every word was an isolated symbol with no relationship to any other.

This lesson fixes that. By the end you will know what an embedding is, how similarity between words gets measured with arithmetic, where the numbers come from, and what the word “dimension” means when somebody says a model uses 768 of them.

Arithmetic used: multiplication, addition, and one square root. All of it shown worked out.

1. The constraint everything else follows from

A neural network multiplies numbers and adds them up. That is the entire operation. It cannot take the word “cat” as input, because “cat” is not a number.

So the first decision in building any of these systems is: how do you turn the thing you care about into numbers? That decision constrains everything downstream, because whatever structure you put into the numbers is the only structure the model can ever find. Anything you throw away at this step is gone permanently.

Two obvious approaches come to mind before the right one. Both are worth working through, because the reason each fails is the reason the real answer looks the way it does.

2. First attempt: number the words

Build a vocabulary and assign each word an ID.

the        -> 0
cat        -> 1
dog        -> 2
sat        -> 3
elephant   -> 4
tuesday    -> 5

Now “the cat sat” becomes [0, 1, 3]. Numbers. Done.

Except this is broken in a way that is easy to miss. You have just told the network that tuesday (5) is five times cat (1), that dog (2) sits exactly halfway between cat (1) and sat (3), and that elephant (4) and tuesday (5) are neighbours.

None of that is true. The IDs were assigned in whatever order the words happened to appear. But the network has no way of knowing they are arbitrary labels. It multiplies and adds, so it will multiply and add these, and it will find patterns in numerical relationships that mean nothing at all.

This is the same reason you do not encode countries as france=1, germany=2, japan=3 in any statistical model. The encoding invents an ordering and a scale that were never in the data.

There is one exception worth noting so you are not confused later: this trick is fine when the categories genuinely have an order and even spacing, like star ratings from 1 to 5. Words do not.

3. Second attempt: one-hot encoding

Fix the invented ordering by giving each word its own slot. With six words in the vocabulary, each word becomes a list of six numbers, all zero except a single 1 in that word’s position.

the        -> [1, 0, 0, 0, 0, 0]
cat        -> [0, 1, 0, 0, 0, 0]
dog        -> [0, 0, 1, 0, 0, 0]
sat        -> [0, 0, 0, 1, 0, 0]
elephant   -> [0, 0, 0, 0, 1, 0]
tuesday    -> [0, 0, 0, 0, 0, 1]

This is called one-hot encoding, because exactly one position is “hot”.

It solves the problem from section 2 completely. There is no ordering, no scale, no accidental arithmetic relationship. Every word is genuinely distinct.

It is also honest in a way worth appreciating: it makes no claim about which words are similar. And that turns out to be the problem.

What one-hot costs

Every pair of words is exactly as different as every other pair. Not approximately. Exactly. Measure the straight-line distance between any two of those vectors and you get the same answer every time, √2, about 1.414. The distance between cat and dog is identical to the distance between cat and tuesday.

So the model gets told that cat and dog have nothing more in common than cat and tuesday. Which is the exact failure from lesson 1, now written in a more sophisticated notation.

The size is absurd. With a realistic vocabulary of 50,000 words, every single word is a list of 50,000 numbers, 49,999 of which are zero. If you stored the full table as a matrix of 32-bit floats, that is 50,000 × 50,000 × 4 bytes, which comes to 10 gigabytes to represent a vocabulary that fits in a small text file.

You have spent 10 GB to encode nothing except “these words are different from each other”, which you already knew.

4. The fix: embeddings

Instead of 50,000 numbers per word, mostly zeros, use a short list of numbers per word, all of them meaningful. Say 300 numbers instead of 50,000.

cat        -> [0.90, 0.20, -0.31, ... 297 more]
dog        -> [0.80, 0.30, -0.28, ... 297 more]
elephant   -> [0.70, 0.35, -0.19, ... 297 more]
tuesday    -> [0.10, 0.95,  0.62, ... 297 more]

This is an embedding. The list of numbers for one word is that word’s embedding vector, and the length of the list is the embedding dimension. When documentation says a model has a hidden size of 768, that is the embedding dimension: every token is represented by 768 numbers.

The size problem disappears immediately. 50,000 words × 300 numbers × 4 bytes is 60 megabytes, instead of 10 gigabytes. Same vocabulary, 0.6% of the memory.

But size is the smaller win. The real win is that these vectors can now be close to each other.

What do the individual numbers mean?

Nothing, individually. This is worth being clear about because it is a common source of confusion.

There is no dimension that means “animalness” and no dimension that means “formality”. Nobody assigned meanings, and if you print out dimension 47 across a real model’s vocabulary you will find no interpretation that holds up. Occasionally a researcher finds a direction that correlates with something recognisable, and it makes for a nice paper, but it is the exception.

What carries the information is where each word sits relative to the others. Words used in similar ways end up near each other. That arrangement is the whole content of an embedding, and it is enough.

5. Measuring similarity

If “near each other” is the point, you need a way to measure nearness. Use two dimensions so it can be checked by hand.

cat        [0.90, 0.20]
dog        [0.80, 0.30]
elephant   [0.70, 0.35]
tuesday    [0.10, 0.95]
wednesday  [0.15, 0.90]

Plot those on paper. Cat, dog and elephant cluster toward the right. Tuesday and wednesday cluster toward the top. Two groups.

The standard measurement is cosine similarity. It is the angle between two vectors, ignoring their lengths. Two vectors pointing the same direction score 1, at right angles score 0, opposite directions score -1.

The formula, then a translation:

cosine(a, b) = (a · b) / (|a| × |b|)

a · b is the dot product: multiply the vectors position by position and add up the results. For cat and dog:

0.90 × 0.80 = 0.72
0.20 × 0.30 = 0.06
                 ----
dot product =    0.78

|a| is the length of a vector, found with Pythagoras: square each number, add them, take the square root.

|cat| = √(0.90² + 0.20²) = √(0.81 + 0.04) = √0.85 = 0.922
|dog| = √(0.80² + 0.30²) = √(0.64 + 0.09) = √0.73 = 0.854

Divide:

cosine(cat, dog) = 0.78 / (0.922 × 0.854) = 0.78 / 0.787 = 0.990

Dividing by the lengths is what makes this a measure of direction only. A vector twice as long pointing the same way scores identically.

Every pair, computed by the script at the end of this lesson:

The animals score above 0.97 with each other. The days score 0.998 with each other. Across the groups, everything drops to 0.3 to 0.6.

Compare that against one-hot, where every one of those pairs would have scored 0.000. That difference is the entire point of this lesson. The model can now be told that cat and dog are related, using nothing but arithmetic it was already doing.

Why 300 dimensions and not 2

Two dimensions can only express one kind of similarity at a time. Real words are similar along many independent axes at once: cat is like dog in being an animal, like chair in being a common concrete noun, like cot in spelling, like tiger in being a feline. Squeeze all of that into a flat plane and the arrangement becomes impossible, because pulling cat toward tiger drags it away from chair.

More dimensions means more room to satisfy many constraints simultaneously. High dimensional space is extremely roomy, in ways that stop matching physical intuition quite quickly. Typical values run from 300 for older word embeddings up to 4,096 or more inside large models.

You cannot picture it. Nobody can. Work with the arithmetic instead and it behaves fine.

6. Where the numbers come from

I have been showing hand-picked vectors. Real ones are not chosen by anyone.

The embedding table is a parameter. Remember from lesson 1 that parameters are adjustable numbers set by training rather than by a person. A 50,000 word vocabulary with 300 dimensions is a table of 15 million numbers, and all 15 million of them start out random and get adjusted.

Which means that at the start of training, all the vectors sit in random positions, cat is nowhere near dog, and the arrangement means nothing. The clustering shown above is a result of training, not an input to it.

How training produces the clustering

Lesson 4 covers the mechanism properly. The idea in one paragraph:

The model is trained to predict words from their surroundings. Both cat and dog appear before “barked”, “slept”, “is hungry”, “vet”. To make good predictions after both of them, the model finds it useful to give them similar vectors, because similar inputs produce similar outputs. Words appearing in different contexts get pushed apart, because keeping them apart makes predictions better. Nobody tells the model that cats and dogs are both animals. The arrangement falls out of the prediction task.

This rests on an old idea from linguistics: a word’s meaning is largely captured by the contexts it turns up in. Words that keep the same company end up meaning similar things. That principle is why this works at all, and it is also why embeddings capture how words are used rather than what they are.

The lookup is a matrix multiplication

Worth seeing, because it explains a piece of notation you will meet constantly.

Stack all the embedding vectors into a matrix, one row per word. Take a one-hot vector for elephant, which is row index 2 in the earlier five word example, and multiply:

[0, 0, 1, 0, 0]  ×  the 5-row matrix  =  [0.70, 0.35]

Verified output from the script:

one-hot @ matrix = [0.7  0.35]
row 2 directly   = [0.7  0.35]

Multiplying by a one-hot vector selects a row. That is all it does. Every zero kills its row, the single 1 keeps its row.

So one-hot encoding never disappeared. It is still conceptually the input format, and the embedding table converts it into something dense. In practice no framework performs that multiplication, because looking up row 2 directly is faster and gives the same answer, but the notation in papers is written as a matrix multiply and now you know why.

7. Two things that are commonly overstated

Vector arithmetic

You will run into the claim that king - man + woman lands near queen. It does, in some embedding sets, and it demonstrates something real: consistent relationships between words show up as consistent directions in the space.

The honest version is that it works for a small number of well chosen examples and fails on most others. The comparison usually excludes the input words from the results, which flatters it considerably. It is a good illustration of what the geometry can do, and a poor summary of what these systems reliably do.

Static embeddings and the word “bank”

The embeddings described so far assign one vector per word, fixed. Look up “bank” and you get the same 300 numbers regardless of context. These are called static embeddings, and word2vec and GloVe are the two names you will see.

Consider:

I sat on the river bank.
I deposited the cheque at the bank.

One vector has to serve both. What training produces is a compromise vector sitting between the two meanings, which is not quite right for either sentence.

Modern language models solve this by producing a contextual embedding: the vector for a word is computed from the whole surrounding sentence, so “bank” in the first sentence gets different numbers from “bank” in the second. The static embedding table is still there, as the starting point, and the model transforms it as the text passes through the layers.

The mechanism that does the transforming is attention, and it is lesson 9. Keep this example in mind until then, because it is the cleanest illustration of what attention is for.

8. Images

Images arrive already numeric, which makes the first step easier and the second step harder.

A greyscale image is a grid of brightness values, usually 0 to 255. A 28 × 28 image of a handwritten digit is 784 numbers:

  0   0   0  12 180 255 190  20   0
  0   0  95 250 130  40 210 160   0
  0  60 240  35   0   0 180 200   0

A colour image has three of these grids stacked, one each for red, green and blue. Those are called channels. A 1024 × 1024 photograph is 1024 × 1024 × 3, which is 3.1 million numbers.

Since it is already numbers, feed it straight in. But raw pixels have the same weakness the word IDs had, in a different form:

• Shift every pixel one position to the right. Nearly identical picture, completely different numbers.
• Brighten the photo. Same scene, every number changed.
• Two pictures of different cats have almost nothing in common at the pixel level.

Numerical closeness in pixel space does not correspond to similarity in content. So images need learned representations for the same reason words do, arrived at from the opposite direction: words need to become numbers, images need their numbers rearranged into something meaningful.

That rearrangement is what convolutional networks did historically and what the early layers of a vision transformer do now. Part 5 covers it properly, including the specific version diffusion models rely on, where an image gets compressed into a compact vector that works much like a word embedding.

9. The code

Everything above, runnable.

import numpy as np

words = ["cat", "dog", "elephant", "tuesday", "wednesday"]
vectors = np.array([
    [0.90, 0.20],
    [0.80, 0.30],
    [0.70, 0.35],
    [0.10, 0.95],
    [0.15, 0.90],
])

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

for i in range(len(words)):
    for j in range(i + 1, len(words)):
        print(f"{words[i]:<10} {words[j]:<10} {cosine(vectors[i], vectors[j]):.3f}")

one_hot = np.array([0, 0, 1, 0, 0])
print("one-hot @ matrix =", one_hot @ vectors)
print("row 2 directly   =", vectors[2])

Notes on the unfamiliar parts:

• np.array builds a matrix from a list of lists. Each inner list is one row, so vectors[2] is elephant’s vector.
• np.dot(a, b) is the dot product from section 5: multiply position by position, add up.
• np.linalg.norm(a) is the vector length, the Pythagoras step.
• @ is Python’s matrix multiplication operator.
• f"{words[i]:<10}" is an f-string, which inserts a value into text. The :<10 pads it to ten characters so the columns line up.
• Install numpy with pip install numpy if you do not have it.

Takeaway

Neural networks only do arithmetic, so words have to become numbers. Numbering them invents relationships that do not exist, and one-hot encoding avoids that but makes every pair of words equally unrelated and wastes enormous space. Embeddings give each word a short list of numbers, learned during training, arranged so that words used in similar contexts sit near each other. Similarity between them is measured with cosine similarity, which is arithmetic the network is already doing. Looking up an embedding is mathematically the same as multiplying a one-hot vector by the embedding table.

Exercises

1. Do the arithmetic by hand. Compute the cosine similarity between cat [0.90, 0.20] and tuesday [0.10, 0.95] on paper, showing the dot product, both lengths, and the division. Check it against 0.318 from the table. Getting this wrong once and finding the error is worth more than reading section 5 twice.

2. Break the geometry. Add a sixth word, kitten, and pick a 2D vector for it that scores above 0.99 with cat. Now add calendar, which should score high with tuesday and wednesday. Then try to add birthday, which is genuinely related to both the days and to nothing else in the animal group. Find out whether you can position it sensibly in two dimensions. Write down what goes wrong. That difficulty is the argument for high dimensional spaces.

3. Prove the lookup claim. Write a loop that multiplies each of the five possible one-hot vectors by the matrix and checks the result equals the corresponding row. Then time both approaches on a 50,000 × 300 random matrix using time.perf_counter and compare. The size of the gap explains why no framework does the multiplication.

4. Use real embeddings. Install gensim and load a pretrained set of word vectors, or use any embedding API. Find the nearest neighbours of “bank”, “python”, and “apple”. Each of those words has two distinct meanings, so look at whether the neighbour list mixes both, and which meaning dominates. That mixing is the static embedding problem from section 7, visible in real data.

5. Connect it back. In lesson 1, the counting model gave “the elephant” a probability of exactly zero. Explain in three or four sentences how embeddings make a non-zero probability possible, given that the model still never saw that pair in training. If you can write this clearly, both lessons landed.