Model parallelism (tensor, pipeline): Foundations

Before you read the parent note Model parallelism (tensor, pipeline), you need a mental picture for every symbol it throws at you. This page builds each one from nothing, in an order where each brick sits on the one before it. Nothing here is "obvious" — we earn it.

1. What is a GPU, and why "memory"?

A GPU (Graphics Processing Unit) is a chip that does thousands of small arithmetic operations at once. For us it is simply a worker with a small desk.

The desk surface is its memory — how many numbers it can hold in front of it right now.
A typical desk holds about 40 GB (gigabytes). One GB = one billion bytes; one byte = 8 on/off switches, enough to store one small number.

Why the topic needs this: the entire reason model parallelism exists is that one desk is too small. Everything downstream is about splitting the pile across many desks. See GPU Memory Hierarchy for the real hardware layers.

2. A number, a vector, a matrix

We build the objects the parent slices.

The symbol $R$ (a stylised "R") means "the set of all ordinary decimal numbers" (R for Real). When the parent writes

$X \in R^{b \times d_{in}}$

read it out loud as: " $X$ is a grid of real numbers with $b$ rows and $d_{in}$ columns." The little " $\in$ " is just the word belongs to. The superscript " $b \times d_{in}$ " is the shape — how tall and how wide the grid is.

Figure — Model parallelism (tensor, pipeline)

Why the topic needs this: every weight in the network is a matrix, and "cutting a matrix into column slices" only makes sense once you can see the grid.

3. The symbols $b$ , $d_{in}$ , $d_{o u t}$ , $d_{m o d e l}$

These are just names for how wide things are. Subscripts (the small letters below) are labels, not multiplication.

Symbol	Plain words	Picture
$b$	batch — how many examples we process at once	number of rows in the input
$d_{in}$	input width — how many numbers describe each example coming in	columns of the input
$d_{o u t}$	output width — how many numbers come out	columns of the output
$d_{m o d e l}$	the standard width used all through a Transformer Architecture	the "main pipe diameter"

Why the topic needs this: the memory saving is always written as a fraction like $d_{o u t} / N$ . You can't feel that saving unless you know $d_{o u t}$ is a width being chopped.

4. Matrix multiplication $Y = X W$

This is the one operation the whole topic is built on, so we define it fully.

Example dot product of $[1, 2, 3]$ and $[4, 5, 6]$ : $1 \cdot 4 + 2 \cdot 5 + 3 \cdot 6 = 4 + 10 + 18 = 32.$

Why multiply-then-add? Each output number is a weighted vote: the input values ( $1, 2, 3$ ) each get a weight ( $4, 5, 6$ ) saying how much they matter, and we total the votes. That single number is one neuron's response.

For the shapes to fit, the inner dimensions must match: $X$ is $b \times d_{in}$ , $W$ is $d_{in} \times d_{o u t}$ , and the shared $d_{in}$ cancels, leaving $Y$ of shape $b \times d_{o u t}$ .

Why the topic needs this: "column-wise split" and "row-wise split" only work because of which dimension of $W$ we cut. You must see the two $d_{in}$ 's meeting in the middle to understand why a column split of $W$ produces a column split of $Y$ .

5. Cutting a matrix: columns vs rows

Now the key move. A matrix can be sliced two ways.

The bar " $∣$ " inside brackets means placed side by side (concatenation). So $[W_{1} ∣ W_{2}]$ means "strip 1, then strip 2, glued together left-to-right."

Why it works (the one fact tensor parallelism leans on): because each output column of $Y = X W$ only uses the matching column of $W$ , cutting $W$ into column strips cuts $Y$ into the same column strips — no strip needs to know about any other. That independence is what lets different GPUs work without talking:

$X [W_{1} ∣ W_{2}] = [X W_{1} ∣ X W_{2}] .$

Why the topic needs this: this is tensor parallelism. Device $i$ keeps only strip $W_{i}$ , and produces only strip $Y_{i}$ .

6. Splitting the work: gather, all-reduce

When each desk only made a piece of the answer, we must combine pieces. Two combine-moves appear in the parent.

The symbol $\sum_{i = 1}^{N}$ (a big Greek "S", sum) means "add these up for $i = 1, 2, \dots, N$ ." So

$y = \sum_{i = 1}^{N} y_{i} = y_{1} + y_{2} + \dots + y_{N}$

is exactly an all-reduce written in maths.

Why the topic needs this: communication is the cost of splitting. Column-then-row layout in the MLP is chosen precisely to need only one all-reduce.

7. Forward, backward, gradient, $\partial$

The network learns by a two-way trip.

The symbol $\frac{\partial L}{\partial W}$ (curly-d, called partial derivative) reads: "how much the loss $L$ changes when we jiggle $W$ a tiny bit." Picture: a slope. If jiggling $W$ upward makes $L$ go down steeply, that slope is large and negative, so we push $W$ that way.

Why a derivative and not just subtraction? We can't try every possible weight. The derivative answers a local question — "which tiny direction improves things right here?" — using one measurement instead of a search.

The superscript $X^{T}$ means transpose: flip the grid so rows become columns. It appears in $\frac{\partial L}{\partial W} = X^{T} \frac{\partial L}{\partial Y}$ purely so the shapes line up for a legal matrix multiply.

Why the topic needs this: memory isn't just weights — the backward pass must store activations from the forward pass. That storage is what pipeline schedules (GPipe, 1F1B) fight over. See Activation Checkpointing and Gradient Accumulation.

8. Devices $N$ , $D$ ; micro-batches $M$ , $B$ ; the "bubble"

Final counting symbols.

Symbol	Plain words	Picture
$N$	number of devices sharing one layer (tensor parallel)	strips per matrix
$D$	number of pipeline stages / devices in a line	desks in a row
$M$ (or $B$ )	number of micro-batches — small chunks of one batch	mini-conveyor items

Why more micro-batches ( $M$ ) help: with only one big batch, each desk works once and idles the rest. Chopping the batch into $M$ pieces keeps every desk fed almost continuously, shrinking the bubble — at the cost of storing more activations. This is the central trade-off of pipeline parallelism, and it connects to Data Parallelism (an orthogonal way to use extra devices) and Mixed Precision Training (which shrinks the numbers themselves).

Prerequisite map

Equipment checklist

Cover the right side and test yourself.

What does

M \in R^{3 \times 5}

mean?

A grid of real numbers, 3 rows tall and 5 columns wide.

Y = X W

, which dimensions must match?

The inner ones: the columns of

X

equal the rows of

W

, and they cancel.

How do you compute one entry of

Y = X W

Dot product: one row of

X

times one column of

W

— multiply matching entries and add them.

What is a column split of

W

Cutting

W

into vertical strips

[W_{1} ∣ W_{2} ∣ \dots]

, each with all rows but only some columns.

Why does a column split of

W

give a column split of

Y

Each output column uses only its matching column of

W

, so the strips are independent.

What is the difference between all-gather and all-reduce?

Gather concatenates pieces side by side; all-reduce adds equal-sized partial results into one sum.

What does

\frac{\partial L}{\partial W}

mean in words?

How much the loss changes when you jiggle

W

slightly — the slope telling us how to nudge

W

What is the pipeline bubble and its formula?

Idle time while later devices wait for earlier ones; fraction

\frac{D - 1}{M + D - 1}

Why do more micro-batches

M

reduce the bubble?

They keep every stage fed almost continuously, shrinking idle time — at the cost of storing more activations.

Why does model parallelism exist at all?

One GPU's memory is too small to hold a huge model, so we cut the model across many GPUs.