Byte-pair encoding, vocabulary design, and what an embedding space actually contains.
8h · 7 lessons · 1 challenge
Before this: Sequence Models
Tokenization is where a surprising number of model behaviors originate: why arithmetic is hard, why some languages cost three times more, why a trailing space changes an answer. You implement BPE, so none of that stays mysterious.
The embeddings half goes from word2vec to contextual representations, with the geometry made visible rather than asserted.
Prompts that fail for tokenization reasons alone. Predict the failure, then fix the prompt without changing its meaning.
Worth 400 XP.
