Model parallelism (tensor, pipeline): Question bank

This page hunts the misconceptions that live inside Model parallelism (tensor, pipeline). Each line is a question ::: answer reveal. Cover the question, commit to an answer out loud, then reveal — if your reason doesn't match the reason given, you found a gap.

Before you start, three words we lean on constantly:

True or false — justify

Tensor parallelism reduces the number of parameters the model has.

False. The parameter count is unchanged; the same weights are just sliced across devices so each device stores

1/ N

of them. The model is mathematically identical.

Data parallelism and tensor parallelism solve the same problem.

False. Data Parallelism replicates the whole model and splits the data (helps throughput, not per-device memory). Tensor parallelism splits the model to fit weights that don't fit on one GPU. They are complementary, often combined.

In column-wise tensor parallelism the input

X

is split across devices.

False. For

Y = X W

with

W

column-split, every device needs the full $X$ ; only

W

(and therefore the output

Y

) is sliced.

X

must be broadcast, not partitioned.

Pipeline parallelism can give perfect (100%) device utilisation with a single mini-batch.

False. With one batch and

D

stages the bubble is

\frac{D - 1}{D}

of the time — most devices idle. You need many micro-batches to fill the pipeline.

GPipe and PipeDream (1F1B) differ only in speed, not memory.

False. Their steady-state activation memory differs: GPipe stores activations for all

M

micro-batches; 1F1B keeps only about

D

micro-batches live because it frees activations by running backward passes early.

The MLP block's column-then-row split needs one all-reduce per linear layer.

False. It is designed so the intermediate hidden state stays sliced (no comm after

W_{1}

), and only one all-reduce happens after

W_{2}

— one per MLP block, not per layer.

Increasing the number of micro-batches

M

is always free.

False. Larger

M

shrinks the bubble

\frac{D - 1}{M + D - 1}

but activation memory grows with $M$ (in GPipe), so you eventually run out of memory before the bubble disappears.

Splitting attention across heads requires communication inside each head's computation.

False. Heads are independent, so each device computes its heads fully locally; communication (all-reduce) only comes after the output projection

W_{O}

combines them.

Spot the error

"Column-parallel

W_{1}

then column-parallel

W_{2}

gives the cleanest attention MLP."

Wrong pairing. The efficient pattern is column-parallel $W_{1}$ then row-parallel $W_{2}$ , so the sliced hidden state feeds directly into the row-split second matrix and only one all-reduce is needed.

"After the row-parallel second layer we do an all-gather to build the output."

Wrong collective. Row-parallel outputs are partial sums over the shared output dimension, so they need an all-reduce (sum), not an all-gather (concat).

"To get

\partial L / \partial X

in column-parallel, each device just uses its own slice — no communication."

Wrong.

\partial L / \partial X = \sum_{i} (\partial L / \partial Y_{i}) W_{i}^{T}

; the chain rule sums over all output columns, so you need an all-reduce to add every device's contribution.

"GPipe bubble fraction is

\frac{D - 1}{B + D - 1}

where

B

is the batch size."

Wrong variable. With micro-batching the bubble is

\frac{D - 1}{M + D - 1}

where

M

is the number of micro-batches, not the raw batch size — splitting the batch is the whole point.

"Pipeline parallelism removes the sequential dependency between layers."

Wrong. The dependency is fundamental — layer

k + 1

still needs layer

k

's output. Micro-batching only hides the waiting by overlapping different micro-batches, it doesn't remove the ordering.

"Activation checkpointing lets you raise

M

with zero extra cost."

Wrong. Activation Checkpointing trades memory for compute: it recomputes activations in the backward pass, adding roughly 33% more compute. It reduces memory pressure but is not free.

Why questions

Why does column-wise splitting of

W

produce independent output slices?

Because

X [W_{1} ∣ W_{2} ∣ \dots] = [X W_{1} ∣ X W_{2} ∣ \dots]

— each column block of

W

maps to its own column block of

Y

, so device

i

can compute

X W_{i}

with no knowledge of the other blocks.

Why is the backward pass the one that forces communication in column-parallel layers?

Forward outputs are independent slices, but the input gradient

\partial L / \partial X

mixes all output columns via

W^{T}

, so the per-device contributions must be summed (all-reduce) before flowing to earlier layers.

Why place row-parallel after column-parallel in the MLP?

Column-parallel leaves the hidden state sliced along its columns; a row-parallel matrix consumes exactly that sliced dimension as its rows, so the two fit together with no reshuffle and only a single final all-reduce.

Why does 1F1B (PipeDream) save memory versus GPipe?

By running a backward pass as soon as a micro-batch finishes its forward, it frees that micro-batch's activations early, so only ~

D

micro-batches are alive at once instead of all

M

Why do more micro-batches shrink the pipeline bubble?

With more micro-batches the pipeline stays filled: while stage 0 works on a new micro-batch, later stages are busy on earlier ones, so the fixed fill/drain cost is amortised over more useful work.

Why combine tensor parallelism with Mixed Precision Training and Gradient Accumulation?

They attack different parts of the memory budget: tensor parallelism splits weights across GPUs, mixed precision halves bytes-per-parameter, and gradient accumulation lets a large effective batch fit without storing it all at once.

Edge cases

What does tensor parallelism with

N = 1

reduce to?

The single-device baseline — one "slice" is the whole matrix, no communication, no memory saving. It is the degenerate case that must match ordinary training exactly.

What happens to pipeline utilisation when

M ≫ D

The bubble

\frac{D - 1}{M + D - 1} \to 0

, so utilisation approaches 100% — but activation memory (in GPipe) grows without bound, so memory, not the bubble, becomes the limit.

What is the bubble when

D = 1

(one pipeline stage)?

Zero:

\frac{D - 1}{M + D - 1} = \frac{0}{M} = 0

. With one stage there is nothing downstream to wait for, so pipeline parallelism gives no benefit and no bubble.

In 1F1B, why can early stages see gradients from stale micro-batch versions?

Because forward and backward for different micro-batches interleave, a stage may apply a weight update while later micro-batches were computed on the older weights — a version mismatch that schedules like PipeDream must manage.

If the whole model already fits on one GPU, is model parallelism ever worth it?

Rarely for memory, but it can still help if a single layer's compute is the bottleneck; usually Data Parallelism is preferred first, since model parallelism adds communication overhead you don't otherwise need.

What limits how far you can push tensor parallelism (

N

very large)?

GPU Memory Hierarchy and interconnect bandwidth: each layer needs all-reduces across all

N

devices, so beyond a point communication cost outweighs the memory saved — practical

N

is capped by fast intra-node links.

Recall One-line self-test

Column-parallel forward needs which collective, and backward needs which? ::: Forward needs an all-gather (concat the output slices); backward needs an all-reduce (sum the input-gradient contributions).