RNNs, LSTMs, encoder-decoders and the attention mechanism that replaced them.
10h · 7 lessons · 1 challenge
Before this: Convolutional Networks & Vision
Attention is introduced here for the reason it was invented: the encoder-decoder bottleneck. Meeting it as a fix to a problem you have already felt makes the transformer track land very differently from meeting it as an architecture diagram.
RNNs and LSTMs are covered properly — they still appear in interviews, and truncated backpropagation through time teaches something no feedforward model does.
An alignment heatmap from a translation model. Predict what the model attended to next — then see whether you or it was right.
Worth 450 XP.
