Learning rate schedules, warmup, batch size effects, gradient clipping and mixed precision.
8h · 8 lessons · 1 challenge
Before this: Making Deep Networks Train
The learning rate is the hyperparameter that matters most and the one people tune least methodically. This module covers schedules and warmup, the interaction between batch size and learning rate, clipping, and the precision choices that let large runs fit.
Every claim is checked empirically — you run the sweep and see the shape of the answer.
A fixed compute budget and a model that has not converged. Five runs to find a learning rate and schedule that gets there.
Worth 450 XP.
