Part 2: Language models

Under construction

This arc is being written. The list below is what it will cover, and it may still change.

1. 1TokenizationText is not split into words. Byte pair encoding, subword units, and why "unbelievable" might be three tokens while "the" is one. Vocabulary size as a design tradeoff. Why models are bad at counting letters, and why some languages cost three times more to process.
2. 2The context problemFixed window models, recurrent networks, and why both fall short. What it means for information to have to travel through every intermediate step, and what gets lost along the way. This lesson sets up the exact problem attention solves.
3. 3AttentionThe mechanism, taken slowly. Queries, keys, and values explained through an analogy that actually holds, then dropped in favour of the arithmetic. Worked by hand on a four word sentence with real numbers. Self attention, causal masking, and multiple heads.
4. 4The transformer blockAttention plus a feedforward network plus residual connections plus layer normalisation. What each piece contributes, and what breaks if you remove it. Positional encoding, because attention on its own cannot tell word order. The full forward pass of a small model traced end to end.
5. 5PretrainingThe training objective, the data, and the scale. What a model learns at each stage of training. Scaling laws, and the tradeoff between model size and data size. What "compute" costs in practice. Why the resulting model is strange to talk to.
6. 6Sampling and decodingGreedy decoding, temperature, top-k, top-p, beam search, repetition penalties. Why the same prompt gives different answers, and why that variation lives outside the model rather than inside it. Practical guidance on which settings to use for which task.