Dr Ashish Bamania · Llama Distributed Training

Distributed Training of Llama Explained

How Meta trains at Llama scale using 4D parallelism: FSDP sharding parameters, optimiser states and gradients rather than replicating them; context parallelism splitting 128K+ token sequences; tensor parallelism splitting weights along the hidden dimension; and pipeline parallelism splitting layers into stages fed by micro-batches, with expert parallelism as a fifth dimension for MoE. Grounded in Llama 3-405B's real run - 16,000 GPUs at 80GB HBM3, 54 days, 30.84M GPU-hours - and it gives the arrangement priority order as TP, CP, PP then DP and FSDP, ranked by communication bandwidth.