Distributed training overview: Worked examples

This is a drill page. The parent note gave you the machinery — data parallelism, model parallelism, communication overhead, gradient accumulation. Here we run that machinery on every kind of number it can face: the easy case, the slow-network case, the "no speedup at all" case, the memory-doesn't-fit case, and the exam-trap case where the naive answer is wrong.

Before line one, one promise: every symbol below is re-earned here. If you have not met a term, it is defined the moment it appears.

The scenario matrix

Distributed training problems live in a small number of "cells". If you can solve one example from each cell, you can solve any homework or interview question.

Cell	What makes it special	The trap it tests
A. Ideal / linear	communication time ≈ 0	do you know the best case ceiling?
B. Comm-bound	sync time comparable to compute	speedup collapses below $k$
C. Degenerate $k = 1$	one device only	formula must give speedup exactly 1
D. Limit $k \to \infty$	infinitely many devices	speedup saturates, not $\infty$
E. Memory-bound	model does not fit → must split	pick model parallel, not data parallel
F. Gradient accumulation	batch too big for RAM	simulate large batch, no extra GPU
G. Async / stragglers	one slow worker	sync waits for slowest; async uses stale weights
H. Word problem	plain-English scenario	translate story → the right formula
I. Exam twist	numbers chosen to fool you	"more GPUs" gives less speedup

We now walk nine examples, one per cell, in order.

Every example uses these three quantities, so let us pin them down once, in plain words.

The picture below is the spine of every example. Look at it before reading on.

Cell A — The ideal / linear case

Cell B — The communication-bound case

This is the parent's BERT-Large example, re-derived from scratch.

Worked example Cell B: slow network eats the gains

$T_{compute} = 400$ ms (100 ms per GPU when $k = 4$ ), sync over slow 10 Gb Ethernet costs $T_{comm} = 50$ ms, $k = 4$ .

Forecast: the ideal was $4 \times$ . With 50 ms of talking bolted on, will we keep most of it or lose most of it?

Step 1. Your slice: $T_{compute} / k = 400/4 = 100$ ms. Why this step? Same slicing as Cell A.

Step 2. Denominator: $100 + 50 = 150$ ms. Why this step? Now the talking is not free — it adds directly to each iteration.

Step 3. $S = 400/150 = 2.667$ . Why this step? Old time ÷ new time.

Step 4 (efficiency). Efficiency $= S / k = 2.667/4 = 0.667$ , i.e. 66.7 %. Why this step? Efficiency asks "of the ideal $k \times$ , what fraction did we keep?"

Verify: We paid 50 ms on top of 100 ms of useful work — that is one-third overhead, so we should keep about two-thirds. $0.667 \approx 2/3$ . ✓

Cell C — The degenerate case $k = 1$

Always test a formula on the boundary it must obviously satisfy.

Cell D — The limit $k \to \infty$

The curve shows Cells A–D at once: it rises, then flattens at $S = 8$ . That flat ceiling is why "more GPUs always means faster" is false — the headline mistake from the parent note.

Cell E — The memory-bound case (model does not fit)

Here data parallelism is useless: it needs a full copy on every GPU, but no single GPU can hold one copy.

Worked example Cell E: a model too big for one GPU

A model has $θ = 20 \times 1 0^{9}$ parameters, stored as float32 (4 bytes each). Each GPU has 16 GB of memory. Ignore activations for now. How many GPUs must the model span, and which strategy?

Forecast: will data parallelism help here at all?

Step 1. Memory for weights: $20 \times 1 0^{9} \times 4 bytes = 80 \times 1 0^{9}$ bytes $= 80$ GB. Why this step? Bytes = params × bytes-per-param. This is the raw footprint.

Step 2. Compare to one GPU: $80 GB > 16 GB$ . It does not fit. Why this step? Data parallelism requires one whole copy per device — impossible here.

Step 3. Minimum GPUs to hold the split model: $⌈ 80/16 ⌉ = 5$ . Why this step? We split the parameters across devices (model parallelism). Need enough GPUs so the total memory ≥ 80 GB; $5 \times 16 = 80$ GB just fits.

Verify: $4$ GPUs give $64$ GB $< 80$ (fails); $5$ GPUs give $80$ GB $\geq 80$ (fits). ✓ So the answer is model parallelism across at least 5 GPUs, not data parallelism.

Recall When do I reach for model parallelism instead of data parallelism?

Trigger for model parallelism ::: when a single copy of the model does not fit in one device's memory.

Cell F — Gradient accumulation (big batch, small memory)

Worked example Cell F: simulate a large batch on one GPU

You want an effective batch size $B = 64$ , but your GPU only fits a micro-batch of $b = 8$ . How many accumulation steps $k$ , and what must you divide by?

Forecast: how many forward-backward passes before one weight update?

Step 1. Number of micro-batches: $k = B / b = 64/8 = 8$ . Why this step? You process 8 chunks of 8 samples to cover all 64.

Step 2. Accumulate: sum the 8 micro-gradients, then divide by 8: $g_{accum} = \frac{1}{8} \sum_{i = 1}^{8} g_{i}$ Why this step? Gradients are linear (average of averages = grand average), so this equals the gradient of one batch of 64.

Step 3. Update once, after all 8 passes: $θ \leftarrow θ - η g_{accum}$ . Why this step? Updating each micro-batch would be batch-8 training, not batch-64.

Verify (linearity numeric check). Take 8 numbers $1, 2, \dots, 8$ as toy "gradients". True mean of all $= 4.5$ . Split into pairs ${1, 2}, {3, 4}, {5, 6}, {7, 8}$ , take each pair's mean ${1.5, 3.5, 5.5, 7.5}$ , then average those four $= 4.5$ . ✓ Same answer — accumulation is exact, only slower.

Cell G — Synchronous vs. asynchronous (the straggler)

Worked example Cell G: one slow worker

Four workers finish an iteration in $90, 95, 100, 300$ ms (the last is a straggler). How long is one synchronous iteration, and what does async do differently?

Forecast: in sync mode, does the fast trio wait?

Step 1 (sync). Sync SGD makes everyone wait for the all-reduce, which cannot start until the slowest worker is done. Iteration time $= max (90, 95, 100, 300) = 300$ ms. Why this step? "Lockstep" means the group moves at the pace of its slowest member.

Step 2 (async). Async SGD lets each worker push its update the instant it finishes, using whatever weights it last pulled. Throughput is governed by the average, not the max: mean $= (90 + 95 + 100 + 300) /4 = 146.25$ ms per update. Why this step? Nobody waits, so the fast three keep producing updates.

Step 3 (the cost). Async pays with stale gradients: a worker may compute on $θ_{t}$ but apply its update after others already moved to $θ_{t + 5}$ . Why this step? No free lunch — no waiting means no guarantee everyone sees the latest weights.

Verify: sync = $300$ ms (dominated by straggler), async avg $= 146.25$ ms ( $< 300$ ). ✓ Async is faster in wall-clock but risks convergence quality.

Cell H — Word problem (translate the story)

Worked example Cell H: the manager's question

"We train on 1 GPU in 20 hours. Compute is 400 ms/iter, sync is 50 ms over our current network. If I buy 4 GPUs, how many hours will training take? And is it worth buying 4 more (8 total)?"

Forecast: guess the 4-GPU time before computing.

Step 1 (4 GPUs). From Cell B, $S_{4} = 400/150 = 2.667$ . Why this step? Same numbers as Cell B — the story is Cell B in disguise.

Step 2. New wall-clock: $20 h /2.667 = 7.5$ h. Why this step? Speedup divides the total time.

Step 3 (8 GPUs). $S_{8} = 400/ (400/8 + 50) = 400/ (50 + 50) = 400/100 = 4$ . Time $= 20/4 = 5$ h. Why this step? Plug $k = 8$ into the master formula.

Step 4 (worth it?). Going $4 \to 8$ GPUs (double the cost) cut time only $7.5 \to 5$ h — a $1.5 \times$ improvement, not $2 \times$ . Why this step? We are climbing the flat part of the Cell-D curve; diminishing returns.

Verify: $S_{8} / S_{4} = 4/2.667 = 1.5$ , and $7.5/5 = 1.5$ . ✓ Consistent. Recommend: fix the network first (Cell I shows why).

Cell I — The exam twist (the naive answer is wrong)

Worked example Cell I: "which is faster — 4 fast GPUs or 8 slow-network GPUs?"

Config X: $k = 4$ , fast interconnect, $T_{comm} = 5$ ms. Config Y: $k = 8$ , slow interconnect, $T_{comm} = 60$ ms. $T_{compute} = 400$ ms in both. Which trains faster?

Forecast: more GPUs (Y has 8) sounds faster. Is it?

Step 1 (X). $S_{X} = 400/ (400/4 + 5) = 400/ (100 + 5) = 400/105 = 3.81$ . Why this step? Master formula, $k = 4$ , cheap talking.

Step 2 (Y). $S_{Y} = 400/ (400/8 + 60) = 400/ (50 + 60) = 400/110 = 3.636$ . Why this step? Master formula, $k = 8$ , expensive talking.

Step 3 (compare). $3.81 > 3.636$ , so the 4-GPU box wins despite having half the GPUs. Why this step? Config Y's fat $T_{comm} = 60$ ms swallows the benefit of extra devices — exactly the Cell-D wall biting early.

Verify: $S_{X} - S_{Y} = 3.81 - 3.636 = 0.174 > 0$ . ✓ Fewer GPUs + fast network beats more GPUs + slow network. This is the parent note's headline mistake made concrete.

Recall

Recall Cell A vs Cell D: what caps speedup?

Cell A (free comms) ceiling is $k$ ; Cell D (fixed comms) ceiling is ::: $T_{compute} / T_{comm}$ , reached as $k \to \infty$ .

Recall Which cell forces model parallelism?

Cell E ::: when one full model copy exceeds a single GPU's memory.

Recall Sync vs async iteration cost with a straggler?

Sync ::: pays the slowest worker's time (max); async ::: pays roughly the average but risks stale gradients.

See also: 3.9-Gradient-descent-optimizers for the update rule $θ \leftarrow θ - η g$ , 4.2-GPU-acceleration for where $T_{compute}$ comes from, 5.1-Training-at-scale for real hybrid setups, and 3.4.1-Transformers-overview for why the models in Cell E got so large.