Generative AI, From Scratch/Part 1: Foundations

What a generative model actually is

After thiswhat you will be able to doWrite a working generative model, watch it produce sentences it was never shown, and explain precisely why it is bad.

Questionwhat this lesson answersHow can a word-counting model produce sentences it was never shown, and precisely why is it bad?

Not coveredwhat this lesson leaves outWe do not build a neural network, train anything with gradients, or cover diffusion beyond the caveat in section 9.

Where this lesson is going

By the end of it you will have written a working generative model, watched it produce sentences it was never shown, and understood precisely why it is bad. That last part matters most. The two specific ways this model fails are the two problems that every technique in the next twenty three lessons exists to solve. If you understand the failures clearly now, the rest of the course stops being a list of tricks and becomes a sequence of fixes to problems you have already seen with your own eyes.

No mathematics in this lesson beyond division. Some Python at the end, explained line by line.

1. A model is a function with adjustable numbers inside it

Start with what you already know. Ordinary code looks like this:

def shipping_cost(weight):
    if weight < 1:
        return 5
    return 5 + (weight - 1) * 2

You decided the 5 and the 2. You decided where the threshold sits. Somebody worked out the pricing and you typed it in.

A model looks like this:

def shipping_cost(weight):
    return a * weight + b

Same shape, except nobody typed in a and b. They begin as arbitrary values, maybe 0.3 and -1.7, and then an automatic process adjusts them, over and over, until the function’s answers line up with real examples of weights and costs. Those adjustable values are called parameters, or equivalently weights. Both words mean the same thing and you will see both.

That is the whole idea. A model is a function whose numbers were found rather than written.

When you read that a model has 7 billion parameters, that is a literal count of adjustable numbers inside it. There are 7 billion values like a and b, arranged in a particular structure, and every one of them was set by an automatic adjustment process rather than by a person. Nobody at any company knows what individual parameter number 4,182,663,901 means. It is a number that ended up where it ended up because doing so made the predictions better.

Keep this in mind whenever a model does something impressive or something stupid. There is no hidden rulebook inside it. There is a large pile of numbers and a fixed sequence of arithmetic.

2. Two kinds of model

Discriminative models take data and return a judgement about it.

email text        ->  spam or not spam
photo             ->  cat or dog
transaction       ->  fraudulent or legitimate
x-ray             ->  fracture or no fracture

Generative models produce new data that resembles the data they were trained on.

nothing, or a prompt  ->  a new sentence
nothing, or a prompt  ->  a new image
a few seconds of audio ->  the next few seconds

That is the textbook distinction, and it is useful for about five minutes.

Modern generative models are built almost entirely out of prediction. They are not doing something categorically different from a spam classifier. A spam classifier picks between two options. A language model picks between fifty thousand options, and then does it again, and again, in a loop. The loop is where generation comes from.

Hold onto that, because it explains a great deal of behaviour that otherwise looks mysterious.

3. The trick: produce one small piece at a time

Nobody has worked out how to produce a coherent paragraph in one shot. What works is producing one small piece, then using everything produced so far to decide the next piece, then repeating.

Trace it:

step 1   input:  "The cat sat on the"
         output: "mat"

step 2   input:  "The cat sat on the mat"
         output: "."

step 3   input:  "The cat sat on the mat."
         output: " It"

step 4   input:  "The cat sat on the mat. It"
         output: " was"

Each output gets appended to the input and the whole thing goes back in. This is called autoregressive generation. Auto meaning self, regressive meaning it operates on its own earlier output.

The model never generates a paragraph. The model answers one narrow question, once: given this text, what comes next? A loop wrapped around the model is what produces the paragraph. The loop is about six lines of ordinary code. It contains no intelligence, no parameters, and no learning.

When you watch a chat assistant type out its answer word by word, that is not a typing animation for effect. That is the loop running, one pass through the model per piece of text, in real time.

Several things follow directly from this that are worth having straight now:

• The model cannot plan ahead in any structural sense. At the moment it emits the first word of a sentence, it has not chosen the last one.
• A long answer costs more than a short one in exact proportion to its length, because it is that many more passes through the model.
• Anything the model has already produced becomes part of its input. A mistake early in an answer is now evidence, sitting there in the context, influencing everything after it. This is a genuine and specific failure mode, and it explains why a model that goes wrong tends to keep going wrong in the same direction.

4. The output is a probability distribution, not a word

This is the most important idea in the lesson. If you take one thing away, take this.

The model does not output “mat”. It outputs a score for every word it knows, all at once, in a single pass. If its vocabulary has 50,000 entries, the output is 50,000 numbers, and they are arranged to add up to exactly 1.

Given "The cat sat on the", the output might be:

Every word in the vocabulary gets a number, including ridiculous ones. “Tuesday” is not excluded, it is merely given a probability so small it will effectively never be picked. There is no filtering step and no list of allowed words. Everything is possible and almost everything is vanishingly unlikely.

Numbers arranged like this, non-negative and summing to 1, are called a probability distribution. When people say a model is “confident”, they mean the distribution has one large value and the rest are small. When it is “uncertain”, the probability is spread across many words. This is measurable, and in lesson 3 you will measure it.

The choosing step is separate from the model

Something has to turn those 50,000 numbers into one actual word. That something is not part of the model. It is a separate step, usually a few lines of code, and it is called decoding or sampling.

The model itself is completely deterministic. Same input, same parameters, same 50,000 numbers, every single time. There is no randomness inside it.

So where does the variation come from when you ask the same question twice and get different answers? From the decoding step. If that step always takes the highest number, you get the same answer every time. If it picks randomly in proportion to the probabilities, you get variation.

You will see this demonstrated with real output in a moment, and lesson 12 covers the full set of decoding strategies. For now, hold the separation clearly in your head: the model produces possibilities, the decoder chooses among them. Almost every knob you have ever seen exposed in an API, including temperature, belongs to the decoder rather than the model.

5. Building the smallest possible generative model, by hand

Enough description. Build one.

Here is the entire training corpus:

the cat sat on the mat
the cat ate the fish
the dog sat on the rug

The training procedure is: go through the text and count which word follows which.

Do the word “the” by hand. Find every occurrence and write down what came next:

Six occurrences. Turn the counts into probabilities by dividing each by the total:

That table is a generative model. It was trained by counting. It takes a word and returns a probability distribution over what comes next, which is exactly the job description from section 4. It is called a bigram model, bigram meaning a pair of adjacent words.

Generating from it

Start at “the”. Look up the distribution. Pick from it at random, weighted by probability. Say you land on “dog”. Now look up what follows “dog”, which in this corpus is always “sat”. Then what follows “sat”, which is always “on”. Then “on”, which is always “the”. And now you are back at “the” with five options again.

One possible run:

the -> dog -> sat -> on -> the -> mat

“the dog sat on the mat” is not in the training text. The corpus has “the dog sat on the rug” and “the cat sat on the mat”, and the model has produced a sentence that is neither, by recombining the patterns. That is generation, in miniature, and the mechanism is not different in kind from what a large model does. It is different in degree by an enormous amount, and the rest of the course is about that degree.

6. The same thing in code

Every line explained. Run it.

text = "the cat sat on the mat the cat ate the fish the dog sat on the rug"
words = text.split()

counts = {}
for i in range(len(words) - 1):
    current = words[i]
    next_word = words[i + 1]
    if current not in counts:
        counts[current] = {}
    counts[current][next_word] = counts[current].get(next_word, 0) + 1

print(counts["the"])

What each part does:

• text.split() breaks the string into a list of words, splitting at each space. You get ["the", "cat", "sat", ...].
• counts = {} creates a dictionary. A dictionary is a lookup table: you put things in under a key, and get them back by that key. Here the key is a word, and the value stored under it is another dictionary holding the words that followed it and how many times.
• range(len(words) - 1) counts from 0 up to one before the last position. The minus one is there so that words[i + 1] never runs off the end of the list.
• if current not in counts: counts[current] = {} creates the inner dictionary the first time a word is seen, because you cannot store something inside a dictionary that does not exist yet.
• .get(next_word, 0) looks up the current count for this pair, and returns 0 if the pair has not been seen before. Add 1 to that, store it back.

Running it prints:

{'cat': 2, 'mat': 1, 'fish': 1, 'dog': 1, 'rug': 1}

The table you built by hand, produced by six lines of code.

Turning counts into probabilities

def probabilities(word):
    options = counts[word]
    total = sum(options.values())
    return {w: c / total for w, c in options.items()}

print(probabilities("the"))

Output:

{'cat': 0.333, 'mat': 0.167, 'fish': 0.167, 'dog': 0.167, 'rug': 0.167}

options.values() gives the counts, sum adds them up, and the last line builds a new dictionary with each count divided by the total. The {key: value for ...} shape is called a dictionary comprehension, and it is Python’s compact way of building a dictionary from a loop.

Generating

import random

word = "the"
output = [word]

for _ in range(9):
    options = counts.get(word)
    if not options:
        break
    word = random.choices(list(options.keys()), weights=list(options.values()))[0]
    output.append(word)

print(" ".join(output))

• random.choices picks from a list. The weights argument makes items with higher counts more likely to come out. It returns a list containing one item, which is why [0] is there to pull the item out.
• counts.get(word) returns None if the word was never followed by anything in the training text, which happens for the last word of the corpus. The break handles that case by stopping early.
• " ".join(output) glues the list of words back into a sentence with spaces between them.

Five real runs of that code:

the cat sat on the fish the cat ate the
the mat the cat sat on the cat ate the
the fish the rug
the cat ate the cat sat on the cat sat
the fish the fish the cat sat on the cat

Look at what happened. Some fragments are perfectly reasonable English: “the cat sat on the”, “the cat ate the”. Some are nonsense: “on the fish”, “the fish the fish”. The third run stopped after four words because it hit “rug”, which is the last word in the corpus and therefore has nothing recorded after it.

You have written a generative model. Now find out why it is bad.

7. Demonstrating that decoding is a separate choice

Section 4 claimed that the randomness lives in the decoder rather than the model. Test it. Change one line so that instead of sampling, it always takes the most likely option:

word = max(options, key=options.get)

Same model, same counts, same everything. Only the choosing step changed. The output:

the cat sat on the cat sat on the cat sat on

It locks into a loop immediately and stays there forever. Always taking the highest probability is called greedy decoding, and this degenerate repetition is exactly what it produces, in tiny models and in large ones. Real systems repeat in subtler ways, but the cause is the same.

Two decoders, one model, completely different behaviour. That separation is real and it is worth remembering the first time somebody tells you a model is “creative” or “boring”.

8. Why this model is bad, and why every later lesson exists

Two problems. Almost everything in this course is a response to one or the other.

Problem one: it only remembers one word back

The model’s entire knowledge of the situation is the single previous word. When it is sitting at “the”, it has no idea whether the sentence so far was “the dog chased the” or “the chef seasoned the”. Both look identical to it.

The obvious fix is to look further back. Count triples instead of pairs, or count based on the previous ten words. This works a little and then collapses, for a reason worth working out in numbers.

Take a realistic vocabulary of 50,000 words, which is roughly what GPT-2 used.

• Pairs: 50,000 x 50,000 = 2.5 billion possible entries. Large but storable.
• Triples: 50,000³ = 125 trillion entries. At 4 bytes each that is 500 terabytes, for a model that looks two words back.
• Ten words back: 50,000¹⁰, which is a number with 47 digits. There are roughly 10⁸⁰ atoms in the observable universe, so this is not a hardware problem you can wait out.

And the storage is not even the real issue. Almost every one of those entries would be empty, because no quantity of text contains most ten word sequences. You would have a gigantic table that is blank nearly everywhere, which is worthless for prediction. This is known as the curse of dimensionality, and it is the reason counting cannot be scaled into a real language model.

Problem two: it cannot generalise at all

The training text never contains “the elephant”. So the model assigns “elephant” a probability of exactly zero after “the”. Not small. Zero. It will never produce it.

That is not a lack of data, it is a structural limitation. To this model, “elephant”, “dog”, and “tuesday” are three unrelated symbols with nothing in common. It has learned that “dog” can follow “the”, and there is no mechanism by which that knowledge transfers to any other animal, or to any other noun, or to anything at all.

Human learning is not like this. You have never read the sentence “the elephant sat on the rug” either, but you know it is fine English, because you know an elephant is the kind of thing that sits and a rug is the kind of thing that gets sat on. The counting model has no representation in which that similarity could exist.

How neural networks fix both

Preview, so you know where the next five lessons are heading.

For problem one, stop storing a table and start computing the answer. A neural network has a fixed number of parameters that does not grow when you feed it longer contexts. A model with 100 million parameters can take 1,000 words of context, and it does not need 50,000¹⁰⁰⁰ of anything. Lessons 4, 5 and 6 cover this.

For problem two, stop treating words as unrelated symbols. Represent each word as a list of numbers, positioned so that words used in similar ways end up with similar lists. Then “elephant” sits near “dog”, and whatever the model learned about how “dog” behaves partly applies to “elephant” automatically, because the model sees similar numbers going in. These lists are called embeddings and they are the entire subject of lesson 2.

Those two fixes, taken seriously and scaled up, get you most of the way to a modern language model. The transformer architecture in Part 2 is a particularly good way of doing the first one.

9. One honest caveat about images

Everything above described autoregressive generation, which is how language models work. Image models mostly do not work this way.

Generating an image one pixel at a time, left to right, was tried and it works badly. A megapixel image would need a million sequential passes, and the result tends to lose coherence across the picture. Image generation is dominated instead by diffusion, which starts from pure random noise and refines the entire image at once, repeatedly, until it becomes a picture.

Different principle, different mathematics, covered properly in Part 5. I am flagging it now so you do not build a mental model where “generative AI” means “predicts the next thing” and then get confused in lesson 22.

Takeaway

A generative model produces one small piece at a time, feeding its own output back in as input. Each step, it outputs a probability across every possible next piece, and a separate decoding step chooses one of them. Counting words gives you a working version of this in six lines of code, and it fails for two reasons: it cannot look back far without the table exploding, and it cannot generalise from one word to a similar one. Everything else in this course exists to solve those two problems.

Exercises

1. Scale it up. Get a large plain text file. Project Gutenberg is the easy source, and any novel will do. Feed it into the counting script, then generate 50 words. Read the output and write down two specific things that are wrong with it. Be concrete. “It does not make sense” is not an answer, “it switches subject halfway through the sentence and never closes a quotation mark” is.

2. Extend to triples. Modify the code so the key is a pair of words rather than one word, and the model predicts the third from the previous two. In Python you can use a tuple as a dictionary key: counts[(word1, word2)]. Generate from it and compare against the bigram output.

Then answer this: it got better, but what did it cost? Print len(counts) for both versions on the same input text and look at the difference. Now imagine going to four words back.

3. Predict the failure. Before running anything, work out what happens if you train on one book and then start generation from a word that appears exactly once in it, as the very last word of the text. Then test whether you were right.

4. Think about the decoder. Modify the sampling so it only ever picks among the top 2 most likely options, ignoring everything else. Run it on the large text. Is the output better or worse than full sampling? There is no single correct answer here, and the tradeoff you are feeling is the subject of lesson 12.