Generative AI, From Scratch/Part 1: Foundations

Probability, only the parts you need

After thiswhat you will be able to doConvert raw scores into a distribution by hand, sample one outcome from it, measure how concentrated it is, and compute the cross entropy and perplexity a model scores on real text.

Questionwhat this lesson answersHow does a network's raw output become a probability distribution, and what single number says whether those probabilities were any good?

Not coveredwhat this lesson leaves outWe do not adjust a parameter to make the score better, cover temperature or the other decoding controls, say where the softmax function comes from, or fix the zero probability a counting model gives a pair it never saw.

Where this lesson is going

Lesson 1 established that a model outputs a probability distribution rather than a word. This lesson takes that apart properly: how raw network output becomes a distribution, how a single number gets picked from one, how to measure whether a model is confident or guessing, and how to score a model’s predictions against reality.

That last one matters more than the rest combined. Training a model means adjusting parameters until a particular score improves, so until you know what the score is, “training” is a word without content. By the end of this lesson you will have calculated that score by hand.

Arithmetic used: multiplication, division, and logarithms. Logarithms get explained from scratch when they show up in section 6, so if you have forgotten them or never met them, you are in the right place.

1. What a probability is here

In this course, a probability is a number between 0 and 1 saying how likely something is. 0 means it will not happen, 1 means it certainly will, 0.25 means it happens roughly a quarter of the time.

The counting model from lesson 1 produced these directly. The word “the” was followed by “cat” twice out of six occurrences, so P(cat follows the) = 2/6 = 0.333. Count the outcome, divide by the total.

Two notations you will see everywhere:

• P(x) means the probability of x.
• P(x | y) means the probability of x given that y already happened. The vertical bar is read as “given”. P(mat | "the cat sat on the") is the probability of “mat” given that specific preceding text. Every language model prediction is a conditional probability of this shape.

A probability distribution is the full set of probabilities across every possible outcome. Two requirements, and both are strict:

1. Every value is zero or greater.
2. All values add up to exactly 1.

The second one is the interesting constraint. It means probability is a fixed quantity being divided up. If the model raises the probability of “mat”, something else has to give up an equal amount. Nothing can go up without something else coming down.

2. Getting from network output to a distribution

A neural network multiplies and adds. The numbers coming out of the final layer are unconstrained: they can be negative, they can be 40, they can be -2.6. They are called logits, and they satisfy neither requirement above.

For four candidate words, the raw logits might be:

mat      3.2
floor    2.1
couch    1.8
bicycle -2.0

Higher means the model prefers it, so the ordering is already meaningful. But you cannot sample from these, cannot compare them across different inputs, and cannot say “the model gave mat a 63% chance”, because -2.0 is not a probability and the four do not sum to anything in particular.

The conversion is a function called softmax. Two steps:

Step one: exponentiate each logit. Raising e (about 2.718) to the power of each number.

e^3.2  = 24.533
e^2.1  =  8.166
e^1.8  =  6.050
e^-2.0 =  0.135

This does two useful things at once. Everything becomes positive, which handles requirement one, including the negative logit that turned into a small positive number. And gaps get stretched: the logit gap between 3.2 and 2.1 was 1.1, while the resulting gap between 24.5 and 8.2 is a factor of three. Exponentiating makes the model’s preferences sharper.

Step two: divide each by the total.

total = 24.533 + 8.166 + 6.050 + 0.135 = 38.884

mat     = 24.533 / 38.884 = 0.6309
floor   =  8.166 / 38.884 = 0.2100
couch   =  6.050 / 38.884 = 0.1556
bicycle =  0.135 / 38.884 = 0.0035

Dividing everything by the total forces the sum to 1, which handles requirement two. Check: 0.6309 + 0.2100 + 0.1556 + 0.0035 = 1.0000.

A real language model does this over the entire vocabulary, so it is 50,000 exponentials and one division per prediction rather than four. The arithmetic is identical.

Two properties worth carrying forward:

• Softmax never outputs an exact zero. Even “bicycle” at 0.0035 could in principle be selected. Nothing in the vocabulary is ever completely ruled out, which is a real difference from the counting model where unseen pairs got a hard zero.
• Only the differences between logits matter, not their absolute size. Adding 10 to all four logits produces exactly the same distribution, because the extra factor cancels in the division.

3. Sampling: picking one

The distribution is a set of probabilities. Turning it into one concrete word is the decoding step from lesson 1, and here is the mechanism.

Line the probabilities up end to end along a number line from 0 to 1, each one taking a stretch as wide as its probability:

0.0        0.6309              0.8409      0.9965  1.0
 |------------|-------------------|-----------|-----|
      mat            floor            couch    bicycle

Those boundaries are running totals: 0.6309, then 0.6309 + 0.2100 = 0.8409, then + 0.1556 = 0.9965, then + 0.0035 = 1.0000. This is called the cumulative distribution.

Now generate a random number between 0 and 1 and see which stretch it lands in.

random value 0.42  -> lands before 0.6309  -> mat
random value 0.73  -> between 0.6309 and 0.8409 -> floor
random value 0.997 -> between 0.9965 and 1.0 -> bicycle

Since “mat” occupies 63% of the line, it gets chosen about 63% of the time. The width of each stretch is exactly its probability, so the selection frequencies match the distribution automatically.

This is all random.choices did in lesson 1, and it is what every sampling method builds on. The variations covered in lesson 12, like top-k and top-p, work by chopping parts off this line and rescaling what remains before drawing.

4. Confidence, measured

“The model is confident” means the distribution is concentrated on few options. “The model is uncertain” means it is spread out. Both statements can be turned into a number.

Compare two distributions over four options:

A:  [0.90, 0.05, 0.03, 0.02]     concentrated
B:  [0.30, 0.30, 0.20, 0.20]     spread out

The measurement is entropy, which is the average surprise of an outcome drawn from the distribution. Low entropy means predictable, high entropy means unpredictable.

The formula, then the translation:

entropy = -Σ p(x) × log₂ p(x)

The Σ means “add up over every option”. Reading it as instructions: for each option, take its probability, multiply by the base-2 logarithm of that probability, add all of those together, flip the sign.

Logarithms are covered properly in the next section. For now the only thing needed is that log₂ of a number below 1 is negative, and it gets more negative as the number gets smaller. That is why the sign gets flipped at the end, to make the result positive.

Computed:

entropy(A) = 0.618 bits
entropy(B) = 1.971 bits

Distribution A is far more concentrated, and its entropy is about a third of B’s.

The upper limit for four options is 2 bits, which happens when all four are equally likely at 0.25 each. B at 1.971 is nearly as uncertain as it is possible to be with four options. The lower limit is 0 bits, when one option has probability 1 and the rest have 0, meaning no surprise at all because the outcome was already certain.

The units are bits because the logarithm was base 2, and a bit here means one yes or no question’s worth of uncertainty. Entropy of 2 bits means you would need about two well chosen yes/no questions to pin down the answer.

Entropy is worth knowing for its own sake, and it also matters because the training objective in section 7 is the same idea applied slightly differently.

5. Likelihood: scoring the model against reality

Everything so far describes what the model predicts. Now the question that training depends on: was the model any good?

The test is straightforward. Take real text the model did not write. At each position, ask what probability the model assigned to the word that actually came next. A good model assigns high probability to what really happened.

Take the sentence “the cat sat” and two models.

Model One:

P(the)            = 0.10
P(cat | the)      = 0.30
P(sat | the cat)  = 0.40

Model Two:

P(the)            = 0.10
P(cat | the)      = 0.02
P(sat | the cat)  = 0.05

Both assigned the same probability to “the”. Model One then expected “cat” and “sat” much more strongly, so it should score better.

To score the whole sentence, multiply the individual probabilities together. Multiplication is right because these are the chances of each step going as it did, and the sentence requires all of them to have gone that way.

Model One: 0.10 × 0.30 × 0.40 = 0.012
Model Two: 0.10 × 0.02 × 0.05 = 0.0001

Model One assigned 120 times more probability to the actual sentence. That number is called the likelihood: the probability the model gives to the data that really occurred.

Training a model means adjusting its parameters to make this number as large as possible on the training data. That is the objective. Everything from here to lesson 6 is machinery for doing it efficiently.

6. Why nobody uses likelihood directly

Two problems, and the fix for both is the same.

The numbers vanish

Three words gave 0.012. A realistic training sequence is 1,000 words, and each factor is well below 1. Multiplying a thousand small numbers together produces something extremely small.

Standard floating point numbers stop at about 1e-308. Below that, the value becomes exactly zero:

0.1 ** 300 = 1e-300      still representable
0.1 ** 400 = 0.0         gone

Once it hits zero the score carries no information at all, and there is no way to tell an improving model from a worsening one. This is called underflow.

Multiplication is awkward to work with

The training procedure in lesson 4 needs to know how the score changes when one parameter is nudged. With a product of a thousand terms, changing one term rescales the entire product. Sums are much easier to reason about and to differentiate.

The fix: logarithms

A logarithm answers the question “what power do I raise the base to, in order to get this number?”

log₁₀(1000) = 3     because 10³ = 1000
log₁₀(0.01) = -2    because 10⁻² = 0.01
log₂(8)     = 3     because 2³ = 8

The base here is usually e, about 2.718, and that version is called the natural logarithm, written ln or just log in most code. The choice of base changes the units and nothing else.

The one property that makes logarithms worth using:

log(a × b) = log(a) + log(b)

Multiplication becomes addition. So instead of multiplying a thousand probabilities, take the log of each and add them. No underflow, because logs of small numbers are moderate negative values rather than vanishingly small positive ones.

For Model One:

log(0.10) = -2.3026
log(0.30) = -1.2040
log(0.40) = -0.9163
                     -------
sum       = -4.4229

Compare that with 0.012. Both express the same thing. One of them survives being extended to a thousand terms.

Log probabilities are always negative, because probabilities are always below 1 and the log of anything below 1 is negative. Closer to zero means better. You will see them constantly, usually called logprobs, and many APIs will return them if asked.

7. Cross entropy: the actual training target

The sum of log probabilities depends on sequence length, so a 1,000 word passage scores much lower than a 3 word one regardless of model quality. Divide by the number of predictions to fix that, and flip the sign so lower is better.

cross entropy = -(sum of log probabilities) / (number of predictions)

For the two models on “the cat sat”:

Model One: 1.474 nats   (2.127 bits)
Model Two: 3.070 nats   (4.429 bits)

This is the number that essentially every language model is trained to reduce. Also called log loss or negative log likelihood, and in code it is usually cross_entropy or nll_loss. Different names, same quantity.

The interpretation is average surprise per prediction. Model Two was more than twice as surprised by real text as Model One was. Training is the process of making the model less surprised by real text, one small parameter adjustment at a time.

“Nats” is the unit when using natural logarithm, “bits” when using base 2. Frameworks report nats by default. The conversion is a fixed factor and neither is more correct.

Note where the units connect back: entropy from section 4 measured surprise using the model’s own distribution, which is uncertainty about its own predictions. Cross entropy measures surprise against what actually happened. Same measurement, different reference point, which is why the names are similar.

Perplexity

One more number, because it appears in every paper. Perplexity is e raised to the cross entropy.

Model One: e^1.474 = 4.37
Model Two: e^3.070 = 21.54

The value of this is that it has a concrete meaning: perplexity is the effective number of options the model was choosing between. Model One behaved as though picking among roughly 4 possibilities at each step. Model Two behaved as though picking among 22.

Check that against a model that knows nothing and spreads probability evenly across a 50,000 word vocabulary. Its perplexity comes out at exactly 50,000, which is the vocabulary size. That is the worst possible score and it lines up with the interpretation: no narrowing down at all.

Lower perplexity means fewer effective options, which means a model that has genuinely narrowed things down. Reported values for good models on ordinary English tend to sit in the low tens or single digits, though the number depends heavily on the tokenizer and the test set, so cross-paper comparisons are less meaningful than they look.

8. The code

import math

# --- softmax ---
logits = [3.2, 2.1, 1.8, -2.0]
exps = [math.exp(x) for x in logits]
total = sum(exps)
probs = [e / total for e in exps]
print("probs:", [round(p, 4) for p in probs])
print("sum:  ", round(sum(probs), 6))

# --- entropy in bits ---
def entropy_bits(p):
    return -sum(x * math.log2(x) for x in p if x > 0)

print("entropy A:", round(entropy_bits([0.90, 0.05, 0.03, 0.02]), 3))
print("entropy B:", round(entropy_bits([0.30, 0.30, 0.20, 0.20]), 3))

# --- likelihood and cross entropy ---
good = [0.10, 0.30, 0.40]
bad  = [0.10, 0.02, 0.05]

def cross_entropy(p):
    return -sum(math.log(x) for x in p) / len(p)

print("good likelihood:  ", math.prod(good))
print("bad  likelihood:  ", math.prod(bad))
print("good cross entropy:", round(cross_entropy(good), 4))
print("bad  cross entropy:", round(cross_entropy(bad), 4))
print("good perplexity:   ", round(math.exp(cross_entropy(good)), 3))
print("bad  perplexity:   ", round(math.exp(cross_entropy(bad)), 3))

Verified output:

probs: [0.6309, 0.21, 0.1556, 0.0035]
sum:   1.0
entropy A: 0.618
entropy B: 1.971
good likelihood:   0.012
bad  likelihood:   0.0001
good cross entropy: 1.4743
bad  cross entropy: 3.0701
good perplexity:    4.368
bad  perplexity:    21.544

Notes:

• math.exp(x) is e to the power of x. math.log(x) is the natural logarithm, math.log2(x) is base 2.
• [math.exp(x) for x in logits] is a list comprehension, which builds a new list by applying something to each item of an old one. Same as a for loop that appends, written compactly.
• math.prod multiplies everything in a list together.
• if x > 0 in the entropy function guards against log2(0), which is undefined. Softmax never produces exact zeros, but hand written distributions might.

Takeaway

A network produces unconstrained logits, and softmax converts them into a probability distribution by exponentiating and dividing by the total. Sampling picks from that distribution by laying the probabilities along a line and drawing a random point. How concentrated a distribution is can be measured as entropy. A model is scored by the probability it assigns to text that actually occurred, which is likelihood, but likelihood underflows to zero on long sequences, so logs are used instead and multiplication becomes addition. Averaging the negative log probabilities gives cross entropy, which is the quantity almost all language model training exists to reduce, and exponentiating it gives perplexity, which reads as the effective number of options the model was picking between.

Exercises

1. Softmax by hand. Compute the softmax of [1.0, 1.0, 1.0]. Predict the answer before calculating. Then do [5.0, 5.0, 5.0] and confirm you get the same result, and explain in one sentence why adding a constant to every logit changes nothing.

2. Sharpen the distribution. Take the logits from section 2 and divide all four by 0.5 before applying softmax, then try dividing by 2.0 instead. Compare the three distributions. One becomes more concentrated and one becomes flatter. That divisor is called temperature and it is the subject of lesson 12, so work out now which direction does which and why.

3. Watch it underflow. Write a loop that starts at 1.0 and multiplies by 0.5 repeatedly, printing the value each iteration. Find the exact iteration where it becomes 0.0. Then run the same loop adding math.log(0.5) instead and confirm it keeps working past that point.

4. Score your lesson 1 model. Take the bigram counting model from lesson 1, trained on a real text file. Hold back a few sentences from training. Compute the cross entropy the model assigns to those held out sentences.

You will hit a problem: some word pairs in the held out text never appeared in training, so their probability is zero, and log(0) is undefined. Do not work around it silently. Write down what this tells you, then look up “add-one smoothing” or “Laplace smoothing” and apply it. Softmax based models never have this problem, and understanding why is worth more than the exercise itself.

5. Interpret a real number. A model reports perplexity 8 on a test set. Say in plain terms what that means about its predictions. Then say what perplexity 1.0 would mean, and why seeing it reported on a test set should make you suspicious rather than impressed.