Part 1: Foundations
The goal of this part is to get you from zero to having written a working neural language model, understanding every line of it.
Read in order
- What a generative model actually isHow can a word-counting model produce sentences it was never shown, and precisely why is it bad?
- Turning words and images into numbersHow do words and images become numbers that preserve useful similarity, and what does dimension mean?
- Probability, only the parts you needHow does a network's raw output become a probability distribution, and what single number says whether those probabilities were any good?
- What training actually meansA loss says how wrong a model is and nothing about what to change. Which of its numbers should move, in which direction, and by how much?
Under construction
This arc is being written. The list below is what it will cover, and it may still change.
- What a neural network layer computesMatrix multiplication as a shape, not a formula. Weights, biases, and activation functions. Why nonlinearity is required, demonstrated by showing what happens without it. Layers stacked, and what depth buys you.
- Building a tiny neural language modelEverything from lessons 1 to 5 assembled into one working program. It predicts the next word from the previous three, it trains, and it gets measurably better than the counting model from lesson 1. Under 100 lines.