Distributed training overview: Question bank — AI-ML deep dive

This is a rapid-fire self-quiz for the parent topic. Every item is a question ::: answer reveal. Read the question, commit to an answer out loud with a reason, then reveal. Bare "yes/no" doesn't count — the whole point of these traps is the justification.

Before you start, a one-line refresher on the vocabulary we lean on:

Recall The three words every item below assumes

Data parallelism ::: every device holds a full copy of the model, but each chews on a different slice of data; gradients are averaged (all-reduced) so all copies stay identical. Model parallelism ::: the model itself is cut into pieces (a layer split = tensor parallel, a group-of-layers split = pipeline parallel); no single device holds all the weights. All-reduce ::: the collective operation that sums (then averages) a tensor across every device and hands the same result back to all of them.

If any of those felt shaky, revisit the parent note before continuing. These prerequisites also help: 3.1-Neural-network-basics, 3.9-Gradient-descent-optimizers, 4.2-GPU-acceleration, 5.1-Training-at-scale.

The two numbers behind every trap

Almost every trap below turns on a tug-of-war between time spent computing and time spent talking. Let's name the quantities in plain words before using them.

Where does the speedup formula come from? Split one training step into two phases and add the times:

Compute phase. The $k$ devices divide the math evenly, so instead of $T_{compute}$ each does $T_{compute} / k$ in parallel — they all finish this phase together after $T_{compute} / k$ seconds.
Communication phase. Then they must sync, costing $T_{comm}$ seconds that nobody can skip.

So one distributed step takes $\frac{T _{compute}}{k} + T_{comm}$ seconds. The single-device baseline takes $T_{compute}$ . Dividing baseline by distributed gives:

$S = \frac{T _{compute}}{\frac{T _{compute}}{k} + T _{comm}}$

Figure s01 — read this first. It shows four horizontal time-bars, one per device count $k \in {1, 2, 4, 8}$ . The lavender segment (compute, $T_{compute} / k$ ) shrinks every time you double $k$ ; the coral segment (communication, $T_{comm}$ ) is identical width in every row. The takeaway to carry away: your total step time can never drop below that fixed coral bar, no matter how many devices you add — that is the ceiling on speedup.

How does $T_{comm}$ itself scale — and does $k$ appear in it? Yes. A realistic all-reduce cost is not just a fixed number; it depends on the message size and on the number of devices:

$T_{comm} \approx latency: hops grow with k α lo g_{2} k + bandwidth term \frac{k - 1}{k} \cdot \frac{M}{β}$

Reading the two pieces:

$α$ is the fixed per-hop hand-shake delay (seconds); a tree/ring all-reduce needs about $lo g_{2} k$ communication rounds, so latency grows slowly with device count.
$M$ is the gradient size in bytes, $β$ is link bandwidth (bytes/second). The factor $\frac{k - 1}{k}$ is the fraction of the data each device must send in an optimal ring all-reduce — it starts at $0$ for $k = 1$ , climbs to $\frac{1}{2}$ at $k = 2$ , and approaches $1$ as $k \to \infty$ (never exceeding it). So the bandwidth cost rises with $k$ but saturates near $M / β$ .

Edge behaviour to notice: at $k = 1$ the whole thing is $0$ (nothing to sync — the $lo g_{2} 1 = 0$ and $\frac{0}{1} = 0$ ). As $k$ grows large, $T_{comm} \to α lo g_{2} k + M / β$ : the bandwidth part flattens out while the latency part keeps creeping up. A faster interconnect raises $β$ , shrinking the bandwidth term — this is exactly what NVLink and InfiniBand buy you.

Figure s02 — read this second. It plots speedup $S$ against device count $k$ . The dashed slate line is the ideal $S = k$ . The three coloured curves feed different $T_{comm}$ values into the speedup formula: mint (fast link), butter (medium), coral (slow). The takeaway: a larger $T_{comm}$ peels the curve away from the ideal line earlier and harder — with a slow link you stop gaining almost immediately, so bandwidth quality decides how far you can usefully scale. These are illustrative curves from the toy formula, not measured hardware numbers.

Formula Linear scaling rule for the learning rate

When data parallelism multiplies the effective batch by $k$ , multiply the learning rate by the same factor: $η_{new} = k \cdot η_{base}$ Why exactly $k$ , not $k$ — the derivation. Think of one SGD step over $k$ small batches done separately versus one big batch of size $k$ done together.

Separate (baseline). Take $k$ ordinary steps, each with the small batch and rate $η_{base}$ . Each step moves the weights by $- η_{base} g_{i}$ where $g_{i}$ is the small-batch gradient. Over $k$ steps (assuming weights barely change so all $g_{i}$ see nearly the same weights) the total move is $- η_{base} \sum_{i = 1}^{k} g_{i}$ .
Combined (large batch). Take one step on the merged batch. Its gradient is the average $\frac{1}{k} \sum_{i = 1}^{k} g_{i}$ , so with rate $η_{new}$ the move is $- η_{new} \cdot \frac{1}{k} \sum_{i = 1}^{k} g_{i}$ .

For the single big step to travel the same distance as the $k$ small steps combined, set the two moves equal: $η_{new} \cdot \frac{1}{k} \sum_{i} g_{i} = η_{base} \sum_{i} g_{i} ⟹ η_{new} = k η_{base} .$ The factor $k$ comes from the averaging in the batch gradient, not from noise. The $k$ figure is a separate fact — it is how much the gradient's noise standard deviation shrinks — and it merely reassures you that the bigger step is safe, but the exact multiplier is $k$ . In practice this is paired with a short warm-up (ramp $η$ up over the first few hundred steps) so the early, unstable phase doesn't diverge.

With the four symbols ( $T_{compute}$ , $T_{comm}$ , $k$ , $S$ ), the speedup formula, and the scaling rule in hand, the traps below should read cleanly.

True or false — justify

Adding more GPUs to a data-parallel job always reduces wall-clock training time.

False. Compute per device shrinks like

T_{compute} / k

, but communication cost

T_{comm}

stays constant or grows (its latency term rises like

lo g_{2} k

); past a point each extra GPU spends more time syncing than computing and total time can rise.

Data parallelism with

k

devices at local batch

b

is mathematically equivalent to single-device training at batch

b \times k

True — but only with a matching learning rate. Averaging gradients across sub-batches equals the full-batch gradient by linearity, yet the effective batch is

bk

, so you usually must scale

η

by the linear scaling rule

η_{new} = k η_{base}

or the optimization dynamics differ.

In pure data parallelism, splitting a model that doesn't fit in one GPU is possible if you use enough GPUs.

False. Every device holds a full copy; if one copy doesn't fit, adding replicas doesn't help. You need model/tensor/pipeline parallelism to split the weights themselves.

Gradient accumulation lets you get the same optimizer update as a large batch without the memory cost.

True. Summing gradients over

k

micro-batches then averaging equals the full-batch gradient; only memory-per-step drops, at the cost of

k \times

more forward/backward passes per update.

Synchronous SGD produces a different model than single-device training.

False. With identical initial weights and averaged gradients, synchronous SGD is deterministically equivalent to single-device SGD on the concatenated batch — that's its main selling point.

Asynchronous SGD converges to exactly the same trajectory as synchronous SGD, just faster.

False. Async workers apply stale gradients (computed on old weights

θ_{t}

but applied to

θ_{t + 5}

), so the optimization path — and sometimes the final accuracy — genuinely differs.

Tensor parallelism and pipeline parallelism are two names for the same thing.

False. Tensor parallelism splits within a layer (a matrix cut into column/row blocks); pipeline parallelism splits across layers (an assembly line of layer groups). They combine, not coincide.

Scaling efficiency above 1 (super-linear) is always a measurement error.

False. It can be real: with more devices each holds a smaller slice, which may now fit in fast cache, giving a per-device speedup that beats the naive

k \times

ideal.

Doubling the interconnect bandwidth doubles your training speedup.

False. Bandwidth only shrinks the

M / β

part of

T_{comm}

; if compute already dominates, halving

T_{comm}

barely moves

S = T_{compute} / (T_{compute} / k + T_{comm})

Spot the error

"We use 4 GPUs data-parallel, local batch 32, so we keep the same learning rate as our batch-32 single-GPU run."

Error: the effective batch is now 128, not 32. The gradient is a much smoother estimate; keeping the old

η

typically underfits or slows convergence — apply the linear scaling rule

η_{new} = 4 η_{base}

(and warm it up).

"Our model is too big for one GPU, so we switched to data parallelism across 8 GPUs."

Error: data parallelism replicates, not splits. A too-big model still won't fit any single replica. The fix is model/tensor/pipeline parallelism (or sharded optimizer states).

"Asynchronous SGD is strictly better because no device ever waits."

Error: it trades waiting for staleness. Removing straggler waits improves hardware utilization but injects stale gradients that can slow or destabilize convergence — "better" is workload-dependent.

"We got 4× speedup on 4 GPUs, so efficiency is over 100%."

Error: 4× on 4 devices is efficiency $= 1$ , i.e. exactly 100%. Efficiency is

S / k = 4/4 = 1

; it would only exceed 100% if

S > 4

"Pipeline parallelism keeps every GPU busy 100% of the time since it's an assembly line."

Error: pipeline bubbles. At the start and end of a batch's journey, downstream stages sit idle waiting for upstream output; without micro-batching these bubbles waste a large fraction of device time.

"All-reduce sends gradients to a central server, which is why it's a bottleneck."

Error: that describes a parameter server, not all-reduce. All-reduce is peer-to-peer (e.g. ring all-reduce) with no central node; its cost grows with data size and device count, not because of one server.

"Since gradient accumulation gives the same update as a big batch, our training finishes just as fast."

Error: same update, not same speed. You still run

k

sequential forward/backward passes per step, so wall-clock time per update is roughly

k \times

longer — you traded memory for time.

Why questions

Why does averaging the per-device gradients (rather than summing them) give the correct large-batch gradient?

Because the full-batch gradient is itself an average over all samples; each device already averages within its sub-batch, so averaging the sub-averages (equal sub-batch sizes) reconstructs the mean over the whole batch by linearity.

Why does scaling efficiency drop as you add more devices?

Compute per device falls as

T_{compute} / k

, but the synchronization cost

T_{comm}

is constant or grows (latency

\sim lo g_{2} k

, bandwidth term climbing toward

M / β

) — so communication becomes a larger fraction of each step.

Why do we synchronize gradients every step in data parallelism instead of occasionally?

To keep all model copies identical; if replicas skipped syncs they would drift into different models, and averaging divergent weights no longer approximates the true update.

Why is high-bandwidth interconnect (NVLink, InfiniBand) worth the cost for large models?

Big models have huge gradient/activation tensors (

M

large), so slow links make the

\frac{k - 1}{k} \cdot M / β

bandwidth term of

T_{comm}

dominate; raising bandwidth

β

shrinks it and recovers most of the lost speedup. The parent note's BERT-Large example (1.3× on 10 Gb Ethernet vs 3.3× on 100 Gb InfiniBand) is a real measured instance of this effect; Figure s02 shows the shape of the same relationship on toy numbers, not those exact measurements.

Why does model parallelism introduce device-to-device transfers that data parallelism (mostly) avoids per layer?

In model parallelism the activations must flow from the device holding one layer to the device holding the next during every forward/backward pass; data parallelism only communicates once per step (the gradient all-reduce).

Why can a straggler cripple synchronous SGD but not asynchronous SGD?

Synchronous SGD is lockstep — everyone waits for the all-reduce, so the slowest device sets the pace; asynchronous workers push updates independently and never block on a slow peer.

Edge cases

What happens to the "effective batch equals

b \times k

" claim if the sub-batches are unequal sizes?

The simple mean of gradients is now slightly wrong; you must take a weighted average by sample count, otherwise small sub-batches get over-weighted relative to their data.

What is the speedup

S

when communication time exactly equals the reduced compute time,

T_{comm} = T_{compute} / k

S = T_{compute} / (2 T_{compute} / k) = k /2

, i.e. only 50% efficiency — half your devices' potential is lost to syncing.

With

k = 1

device, what does the speedup formula

S = T_{compute} / (T_{compute} / k + T_{comm})

give?

With no peers there's nothing to sync, so

T_{comm} = 0

(both the

lo g_{2} 1

and

\frac{k - 1}{k}

terms vanish) and

S = T_{compute} / T_{compute} = 1

— exactly the single-device baseline, as it must.

How does the all-reduce bandwidth factor

\frac{k - 1}{k}

behave as

k

grows without bound?

It rises monotonically from

0

(at

k = 1

) toward

1

but never reaches it, so the bandwidth cost saturates near

M / β

— the latency term

α lo g_{2} k

is what keeps growing after that.

What is the effect of gradient accumulation over

k

micro-batches when

k = 1

It degenerates to ordinary single-batch training — the "sum then divide by 1" is a no-op, so there's no memory saving and no slowdown.

If a model just barely fits on one GPU, is pipeline parallelism across 2 GPUs guaranteed to speed it up?

Not guaranteed — you gain parallel compute but pay activation-transfer cost and pipeline-bubble idle time; for a small model these overheads can outweigh the benefit, giving little or negative speedup.

In tensor parallelism splitting

W

into column blocks

[W_{1} W_{2}]

, what must happen before the next layer if that layer needs the full output?

The partial outputs

Y_{1}, Y_{2}

must be gathered/concatenated (or all-reduced, depending on the split) across devices, adding a communication step between layers.

What limits the maximum useful number of data-parallel replicas for a fixed dataset?

Once each replica's local batch shrinks toward 1 (or the effective batch grows so large it hurts generalization), adding replicas stops helping — you run out of data per step and communication dominates.