Generative AI, From Scratch/Part 1: Foundations

What training actually means

After thiswhat you will be able to doWork out a gradient by hand, check it by nudging, run gradient descent until the loss stops falling, and say what fixes the floor it stops at.

Questionwhat this lesson answersA loss says how wrong a model is and nothing about what to change. Which of its numbers should move, in which direction, and by how much?

Not coveredwhat this lesson leaves outWe do not build the layers being trained, prove the chain rule or any derivative rule, cover Adam and warmup beyond naming what they change, or show how a framework records operations so it can differentiate them.

Where this lesson is going

Lesson 3 ended with a number to reduce: cross entropy. Lesson 4 is the procedure for reducing it.

This is the pivotal lesson of Part 1. Everything before it was setup, and everything after it is architecture. The procedure here, gradient descent driven by backpropagation, is what trains every model in this course, from the four parameter example below to models with hundreds of billions of parameters. Not a simplified version of it. The same procedure.

By the end you will have worked out a gradient by hand, checked it numerically, watched a loss go down step by step, and seen exactly why the training of a language model bottoms out where it does.

Arithmetic used: multiplication and subtraction. Derivatives get built from scratch, and no prior calculus is assumed. If you know calculus already, section 4 will be slow and you can skim it.

1. What we have and what is missing

The situation so far:

• A model is a function with adjustable numbers inside it, called parameters (lesson 1).
• Words become vectors, and the vectors themselves are parameters (lesson 2).
• Cross entropy scores how surprised the model was by real text, lower being better (lesson 3).

What is missing is the connection between them. There are 15 million parameters in a modest embedding table alone. Cross entropy tells you the current arrangement scores 4.2. It does not tell you which of those 15 million numbers to change, in which direction, or by how much.

Training is the answer to that question, and it is more mechanical than its reputation suggests.

2. Why guessing does not work

The obvious approach is to try values and keep what works. Set the parameters randomly, measure the loss, set them randomly again, keep whichever was better.

Put numbers on it. Suppose a model has only 100 parameters, and each one has just 10 plausible values. The number of combinations is 10¹⁰⁰, which is more than the number of atoms in the observable universe. A real model has billions of parameters, each a continuous value.

Random search is not slow here. It is arbitrarily far from possible, and no amount of hardware changes that. Something is needed that does not involve searching.

3. The idea that replaces searching

Rather than asking “what is the best value for this parameter”, ask a much smaller question:

If I nudge this parameter slightly upward, does the loss go up or down?

That question is answerable. Nudge it, recompute the loss, compare. If the loss went down, keep moving in that direction. If it went up, move the other way. Repeat.

This works because you never need to know where the best value is. You only need to know which way is downhill from where you currently stand. Take a small step that way, then ask again from the new position.

The measurement of “which way is downhill and how steeply” is a derivative, and the rest of this lesson is about computing them efficiently.

4. Derivatives, built from scratch

A derivative is a slope: how much the output changes per unit of change in the input.

Work with a concrete loss. Say the model predicts a cost from a weight, using one parameter w:

prediction = w × 2

The correct answer for this example is 6. The loss is the squared difference between prediction and truth, which is a simpler loss than cross entropy and easier to compute by hand. Squaring makes the loss positive whether the prediction was too high or too low.

L(w) = (2w - 6)²

Try w = 1:

L(1) = (2 - 6)² = 16

Now nudge w up by a tiny amount, 0.0001, and recompute:

L(1.0001) = (2.0002 - 6)² = 15.99840004

The loss went down by 0.00159996 for an input change of 0.0001. Divide to get the change per unit:

-0.00159996 / 0.0001 = -15.9996

That is the derivative, approximately. It says: at w = 1, increasing w by 1 would decrease the loss by about 16, if the rate held steady. It does not hold steady, which is why the steps taken later are small.

The sign is the part that matters most. Negative means increasing w decreases the loss, so w should go up.

The exact version

Computing this by nudging is called numerical differentiation, and it gives an approximation. There are rules that give the exact answer directly. For this loss the exact derivative is:

dL/dw = 8w - 24

At w = 1 that gives 8 - 24 = -16, and the nudging method gave -15.9996. The two agree, and the small discrepancy is because 0.0001 is not infinitely small.

The notation dL/dw is read as “the derivative of L with respect to w”, meaning how L changes as w changes. You do not need to memorise derivative rules for this course. What matters is knowing what the number means, and that libraries compute it for you.

Numerical differentiation remains useful as a checking tool. When you write gradient code by hand, comparing it against a nudge is how you find out whether it is right. This is called gradient checking and it is used in section 9.

5. Gradient descent

Now the update rule. Move each parameter in the direction that reduces the loss, by an amount proportional to the slope:

new_w = old_w - (learning_rate × gradient)

The minus sign is there because a negative gradient means “go up” and a positive gradient means “go down”. Subtracting handles both cases.

The learning rate controls step size. Use 0.05 and run it:

The loss drops from 16 to under 0.1 in five steps. w is heading toward 3, which is the value that makes the prediction exactly right, and the gradient shrinks as it gets closer. The steps get smaller automatically near the bottom, because the slope flattens there.

That table is training. A real training run is the same three lines repeated billions of times across billions of parameters: measure the loss, compute the gradients, step downhill.

The usual metaphor is walking downhill in fog. You cannot see the valley floor, but you can feel the slope under your feet, so you step downhill and repeat. It is a decent picture as long as you remember that the hill has billions of dimensions rather than two.

6. The learning rate is the thing that breaks

Same problem, same starting point, learning rate 0.3 instead of 0.05:

The loss increases every step and the parameter oscillates with growing amplitude. The steps overshoot the bottom, land further up the opposite slope, then overshoot harder coming back. This is called divergence, and in a real run it shows up as a loss that climbs and then becomes NaN.

Too small has the opposite failure: the loss decreases correctly but so slowly that training would take months of compute it does not have.

There is no formula for the right value. It is found by trying, typically starting around 0.001 for large models, and it is usually reduced over the course of training on a schedule. If you ever hear that training large models involves a lot of guesswork, this parameter is a large part of what is meant.

7. More than one parameter

Real models have more than one. Add a second:

prediction = w × x + b

Now the loss depends on two numbers. Ask the same question about each one separately: if I nudge w while holding b fixed, what happens to the loss? Then if I nudge b while holding w fixed?

Each answer is a partial derivative, written with a curly d, ∂L/∂w. The only difference from before is that everything else is held still while one thing moves.

With x = 2, correct answer y = 6, and current values w = 1, b = 0:

prediction = 1 × 2 + 0 = 2
error      = 2 - 6 = -4

∂L/∂w = 2 × error × x = 2 × (-4) × 2 = -16
∂L/∂b = 2 × error     = 2 × (-4)     = -8

Collect them into a list: [-16, -8]. That list is the gradient. It has one entry per parameter, and it points in the direction of steepest increase in the loss, so the update subtracts it to go the other way.

For a model with 7 billion parameters, the gradient is a list of 7 billion numbers, recomputed at every single training step. The update rule does not change:

for each parameter:
    parameter = parameter - learning_rate × its_gradient

Note that ∂L/∂w came out twice as large as ∂L/∂b, because x = 2. Parameters that had more influence on the prediction get larger gradients and therefore larger corrections. That falls out of the arithmetic rather than being designed in.

8. Why the nudging method cannot be used

Section 4 computed a derivative by nudging and recomputing. That approach is correct and completely unusable at scale.

Each parameter needs its own nudge and its own loss recomputation. Running the model once is called a forward pass. So:

• 7 billion parameters
• 2 forward passes each, one nudged up and one down
• 14 billion forward passes
• for one training step

Training involves hundreds of thousands of steps. At any realistic speed this finishes long after the sun goes out.

What is needed is a way to get all the gradients from roughly one forward pass instead of billions. That method is backpropagation, and it rests on one rule.

9. The chain rule

The chain rule handles the case where a value flows through several steps.

If changing w changes a, and changing a changes L, then the effect of w on L is the product of the two effects:

dL/dw = dL/da × da/dw

The intuition is unit conversion. If a is 3 times as sensitive as w, and L is 5 times as sensitive as a, then L is 15 times as sensitive as w. Multiply the rates along the path.

Work an example with two parameters in sequence:

a = w1 × x        first step
b = w2 × a        second step
L = (b - y)²      loss

With x = 2, y = 6, w1 = 1.5, w2 = 0.5:

Forward pass, computing left to right and keeping every intermediate value:

a = 1.5 × 2 = 3.0
b = 0.5 × 3.0 = 1.5
L = (1.5 - 6)² = 20.25

Backward pass, computing right to left:

dL/db  = 2 × (b - y) = 2 × (1.5 - 6)   = -9.0
dL/dw2 = dL/db × a   = -9.0 × 3.0      = -27.0
dL/da  = dL/db × w2  = -9.0 × 0.5      = -4.5
dL/dw1 = dL/da × x   = -4.5 × 2.0      = -9.0

Checked against numerical differentiation:

numeric dL/dw1: -9.0
numeric dL/dw2: -27.0

Exact agreement.

Now look at the structure of that backward pass. dL/db was computed once and reused for both dL/dw2 and dL/da. Nothing was recomputed. Every quantity was used and passed further back.

10. Backpropagation

Backpropagation is that backward pass, run over the whole network.

Forward pass. Feed input through the network, computing each layer in turn, storing every intermediate value. End with a loss.

Backward pass. Start at the loss, which has a derivative with respect to itself of 1. Move backwards through the network. At each step, combine the derivative arriving from the layer ahead with the local derivative of the current operation, using the chain rule. Pass the result further back.

The cost is roughly the same as one forward pass, and it produces every gradient for every parameter. That is the difference between 14 billion forward passes and about 2, and it is the reason training large models is possible at all.

Two consequences worth knowing now:

Memory. Every intermediate value from the forward pass has to be kept, because the backward pass needs it. This is why training a model needs far more memory than running one, typically several times more, and why the batch size you can train with is limited by memory rather than compute.

Everything must be differentiable. Each operation in the network needs a local derivative for the chain rule to pass through. This constrains architecture design in ways that show up repeatedly later. When you meet a technique that looks like a strange approximation of something simpler, the usual reason is that the simple version had no usable derivative.

Modern frameworks build the chain of operations automatically as the forward pass runs, then walk it backwards. The feature is called automatic differentiation, and in PyTorch it is what loss.backward() does. You will not write backward passes by hand, and understanding what that call is doing is worth the twenty minutes this section took.

11. The gradient of cross entropy, which is unexpectedly simple

Squared error was used above because it is easy by hand. Language models use cross entropy, and it produces one of the cleanest results in the field.

Take the four logits from lesson 3, and suppose the word that actually came next was “floor”, which is index 1:

logits:  [3.2,    2.1,    1.8,    -2.0  ]
probs:   [0.6309, 0.2100, 0.1556,  0.0035]
correct: floor (index 1)
loss = -log(0.2100) = 1.5606

The gradient of the loss with respect to the logits is:

gradient = predicted probability - correct answer as one-hot

Which gives:

[0.6309 - 0,  0.2100 - 1,  0.1556 - 0,  0.0035 - 0]
= [0.6309, -0.7900, 0.1556, 0.0035]

Checked numerically by nudging each logit:

analytic: [0.6309, -0.79, 0.1556, 0.0035]
numeric:  [0.6309, -0.79, 0.1556, 0.0035]

Identical.

Read what the gradient says. The correct word has a negative gradient, so its logit gets pushed up, and the size of the push is 0.79, which is exactly how much probability it was missing. Every wrong word has a positive gradient, so its logit gets pushed down, and the size is exactly the probability it wrongly took. “Mat” took 0.63 and gets pushed down hardest. “Bicycle” took 0.0035 and is barely touched.

So a single training step on a single token is: raise the score of what actually happened, lower the scores of everything else in proportion to how much they were wrongly favoured. Repeated across trillions of tokens, that is pretraining.

The clean form is not luck. Softmax and cross entropy are used together partly because the messy parts of their derivatives cancel when composed.

12. A complete training run

Everything assembled into the smallest model that is still a language model: a unigram model, which predicts words with no context at all, using four logits as its only parameters.

Training data, a corpus with these word counts:

the  5      cat  3      sat  1      mat  1        total 10

So the frequencies to be learned are 0.5, 0.3, 0.1, 0.1. Start all four logits at zero, which makes the model predict everything equally likely. Learning rate 0.5.

step   0  loss=1.38629  probs=[0.25, 0.25, 0.25, 0.25]
step  60  loss=1.16829  probs=[0.5, 0.299, 0.101, 0.101]
step 120  loss=1.16828  probs=[0.5, 0.3, 0.1, 0.1]
step 180  loss=1.16828  probs=[0.5, 0.3, 0.1, 0.1]
step 240  loss=1.16828  probs=[0.5, 0.3, 0.1, 0.1]

The model learned the word frequencies from nothing but repeated application of the update rule. Nobody counted anything or told it what the answer was.

The loss stopped at 1.168, not at 0

This is worth understanding properly, because it explains what a training curve is doing when it flattens.

That final value is the entropy of the data itself. Compute the entropy of [0.5, 0.3, 0.1, 0.1] using the formula from lesson 3, in nats:

-(0.5 × ln0.5 + 0.3 × ln0.3 + 0.1 × ln0.1 + 0.1 × ln0.1) = 1.16828

The same number, to five decimals.

A loss of zero would require assigning probability 1 to every word that occurs, which is impossible when several different words genuinely occur. The data has real uncertainty in it, and no model can predict away uncertainty that exists in the world. The best possible model reproduces the true distribution exactly, and its loss equals the entropy of that distribution.

So the floor of the loss is set by the data, not by the model. When a language model’s training loss flattens, it has either reached that floor or run out of capacity to get closer to it, and telling those two cases apart is a real practical problem covered in lesson 11.

13. What a real training loop adds

Four differences between the loop above and one that trains a real model.

Batches. Computing the loss over the entire dataset before each step is too slow, and computing it on one example is too noisy. Real training uses a batch of examples, commonly 32 to several million tokens, averages the loss over them, and steps once. This is stochastic gradient descent, stochastic because each batch gives a noisy estimate of the true gradient. The noise turns out to help rather than hurt, since it shakes the parameters out of poor positions.

Epochs. One pass through the training data is an epoch. Small models train for many epochs. Large language models often train for one or less, because the dataset is larger than the compute budget allows repeating.

Better optimizers. Plain gradient descent uses the same learning rate for every parameter forever. Adam and its variants track a running average of each parameter’s recent gradients and scale steps individually, so parameters with consistently small gradients still move. Nearly all large models use Adam or AdamW. The underlying idea is unchanged, and the improvements are refinements of step sizing.

Learning rate schedules. The rate usually starts small, rises over the first few thousand steps, which is called warmup, then decays toward zero. Large steps early, careful steps late.

The skeleton, which is what real training code looks like once the abstractions are stripped away:

for each batch in the data:
    predictions = model(batch.inputs)          # forward pass
    loss = cross_entropy(predictions, batch.targets)
    gradients = backpropagate(loss)            # backward pass
    for each parameter:
        parameter -= learning_rate × gradient  # update

Five lines. A hundred million dollar training run is those five lines, repeated.

14. The code

import math

def softmax(z):
    m = max(z)                       # subtract the max for numerical stability
    e = [math.exp(x - m) for x in z]
    s = sum(e)
    return [x / s for x in e]

words      = ["the", "cat", "sat", "mat"]
counts     = [5, 3, 1, 1]
total      = sum(counts)
empirical  = [c / total for c in counts]

logits = [0.0, 0.0, 0.0, 0.0]        # the parameters
lr = 0.5

for step in range(301):
    p = softmax(logits)                                        # forward pass
    loss = -sum(empirical[i] * math.log(p[i]) for i in range(4))
    if step % 60 == 0:
        print(f"step {step:3d}  loss={loss:.5f}  probs={[round(x,3) for x in p]}")
    grad = [p[i] - empirical[i] for i in range(4)]              # backward pass
    logits = [logits[i] - lr * grad[i] for i in range(4)]       # update

Notes:

• Subtracting the max inside softmax changes nothing mathematically, because section 2 of lesson 3 showed that shifting all logits by a constant leaves the result identical. It prevents math.exp from overflowing on large logits, and every real implementation does it.
• The gradient line is the result from section 11, with the empirical frequencies standing in for the one-hot target because this loss averages over the whole corpus at once.
• Nothing here is a library call. This is gradient descent with no framework at all.

Gradient checking, which is how you verify hand written gradient code:

def numeric_gradient(f, params, i, eps=1e-6):
    up = params[:]; up[i] += eps
    dn = params[:]; dn[i] -= eps
    return (f(up) - f(dn)) / (2 * eps)

Compute the analytic gradient, compute this for each parameter, and compare. If they disagree beyond about 1e-5, the analytic version has a bug.

Takeaway

Searching for good parameter values is impossible at any realistic size, so training instead asks a local question at each parameter: does nudging this up increase or decrease the loss? The answer is a derivative, the full collection of them is the gradient, and subtracting a small multiple of the gradient from every parameter moves the loss downhill. Computing derivatives by nudging would take billions of forward passes, so backpropagation computes all of them in one backward walk using the chain rule. For softmax with cross entropy the gradient with respect to the logits is the predicted probabilities minus the correct answer, so each step raises the score of what actually happened and lowers everything else in proportion to how wrongly it was favoured. The loss cannot reach zero, because it bottoms out at the entropy of the data.

Exercises

1. Feel the learning rate. Take L(w) = (2w - 6)² starting at w = 1 and run gradient descent for 20 steps at learning rates 0.01, 0.05, 0.12, and 0.3. Print w and the loss each step. Find, to two decimal places, the largest learning rate that still converges. Then work out what property of this specific loss function determines that threshold.

2. Check a gradient. Implement the two-layer example from section 9 with your own choice of x, y, w1, w2. Compute dL/dw1 and dL/dw2 by the chain rule, then verify both with the numeric_gradient function. Now deliberately introduce a bug into the analytic version, such as dropping the w2 factor from dL/da, and confirm the check catches it.

3. Extend the unigram model. Modify the section 14 code to learn from an actual text file instead of hardcoded counts. Then compare the trained probabilities against simply counting word frequencies and dividing by the total. They should match closely. Explain why gradient descent is the harder way to get an answer that counting gives instantly, and what changes to make counting stop working.

4. Hit the floor. Run the unigram training with counts [10, 0, 0, 0], so only one word ever occurs. Watch what the loss does over 2,000 steps and what happens to the logits. Explain the behaviour using section 12, and say what would happen in a real model if a token appeared in the vocabulary but never in the training data.

5. Trace a gradient by hand. For the softmax example in section 11, suppose the correct word had been “bicycle” instead of “floor”. Write out the gradient vector without running any code. Which logit gets the largest correction and why? What does that tell you about how much a model learns from a token it found very surprising versus one it nearly predicted?