Data, tensor and pipeline parallelism, ZeRO and FSDP, and making a run survive a failed node.
10h · 8 lessons · 1 challenge
Before this: Optimizers & Training Dynamics
When a model stops fitting on one device you need to know which axis to split and what that costs in communication. This module covers the parallelism strategies, the sharding schemes, and the collective operations underneath them.
Reliability gets real attention: long multi-node runs fail, and the difference between losing an hour and losing a week is entirely preparation.
A model, a cluster and an interconnect. Choose the parallelism split; the game reports whether it fit and what throughput you left on the table.
Worth 650 XP.
