Generative AI, From Scratch

What you will be able to do at the end

• Explain how a large language model works, mechanism by mechanism, without hand waving
• Write a small language model yourself, in Python, and train it
• Fine-tune an existing model, and know when that is the wrong tool
• Build a retrieval system and an agent that are not toys
• Understand what diffusion models do and why they work differently from language models
• Estimate cost, memory, and latency for running a model before you commit to an architecture
• Read a paper about a new technique and place it correctly against everything else

What you need before starting

Python. Enough to run a script and read a for loop. Anything beyond that is explained in the lesson where it appears.

Mathematics. None assumed. No calculus, no linear algebra. The parts you need get built up from scratch, and the notation is always translated into words and code before it is used.

Hardware. A normal laptop is enough for the first three parts. Parts 4 and 5 have optional exercises that want a GPU, and there are free cloud options covered when we get there.

Honest scope

Twenty four lessons. Each one is somewhere between two and five hours if you do the exercises, which means this is a few months of steady part-time work, not a weekend.

The compressed version of this material exists everywhere and it produces people who can talk about attention without being able to implement it. If you want that, you do not need these lessons. This course is written on the assumption that you want to actually understand the thing.

How to work through it

Read the lesson, then do the exercise before moving on. The exercises are short on purpose. They exist because reading about this subject produces a very convincing feeling of understanding that disappears the moment you try to write the code yourself. Finding out where that gap is, in a ten line script, is much cheaper than finding out in lesson 15.

Each lesson ends with a glossary of the terms it introduced. The vocabulary in this field is unusually bad, with several names for the same thing and one name for several things, so the glossaries are worth rereading.

A note on tooling

The course uses Python and PyTorch. NumPy shows up in the early lessons where PyTorch would be overkill. Nothing else is required.

I will name specific models and libraries where it helps, but this field moves quickly and any specific recommendation has a short shelf life. The mechanisms are stable. A transformer works the same way it did in 2017 and the arithmetic in lesson 19 will still be correct in five years. Optimise your attention accordingly.