Part 5: Images, audio, video
The goal of this part is the second major family of generative models, which works on a completely different principle.
Under construction
This arc is being written. The list below is what it will cover, and it may still change.
- Autoencoders and latent spacesCompression as learning. Encoders, decoders, and the compressed representation in the middle. Variational autoencoders and why a well-behaved latent space lets you generate rather than only reconstruct. This is the foundation the next two lessons stand on.
- DiffusionAdd noise to an image until it is pure static, then train a network to undo one step of that. Generate by starting from static and running the undo process repeatedly. The training objective, the sampling loop, and why this beat every earlier approach to image generation.
- Text to imageHow a text prompt steers the denoising process. Joint text and image embedding spaces. Classifier free guidance and what the guidance scale control actually does. Latent diffusion, which is why these models run on consumer hardware at all. ControlNet and image conditioning.
- Audio, video, and what comes nextAudio as a generation problem, covering both speech and music. Video and the temporal consistency problem. Multimodal models that handle several kinds of data in one system. An honest assessment of what is solved, what is not, and how to keep reading the field after this course ends.