Intuition The One Core IdeaTraining a neural network means repeatedly guessing an answer, measuring how wrong the guess is, and nudging millions of internal dials to be a little less wrong next time. Distributed training is simply doing that same nudging with many machines at once — so we must agree, before touching hardware, on what a "dial", a "wrongness score", and a "nudge" actually are .
This page is the ground floor. The parent note Distributed training overview freely writes symbols like θ ∇ θ L η before you meet it again.
Everything in distributed training is a variation on one small loop. Meet it first as a picture, then we name each piece.
Intuition What the loop doesA model looks at data, produces a guess, we score the guess, and we adjust the model. Repeat forever. The distributed question is only: which of these boxes do we copy across machines, and which do we split? To answer that, you must know exactly what lives in each box.
Let us walk the boxes left to right and earn every symbol.
Definition A single training example
x input : a list of numbers describing one thing. For an image it is every pixel brightness laid out in a row; for a sentence it is a list of word-codes.y target : the correct answer we want the model to produce for that x The bold face on x vector . Plain y
Picture x y
D D = {( x i , y i ) } i = 1 N D set (that is what the curly braces { } x i y i i 1 N
i counting index , like a row number.N total number of examples .
Intuition Why we even needD data parallelism means "give different machines different rows of D 3.1-Neural-network-basics for how these inputs feed a network.
θ θ adjustable dials inside the model — every weight and bias. A modern model has billions of these numbers.
When we say "the model", mathematically we mean "the current setting of θ
Picture: a giant mixing board with billions of sliders. Training = sliding them to good positions.
Definition The model functionf y pred = f ( x ; θ ) f x using dial-settings θ
The semicolon ; what changes per example (x what stays fixed during one guess (θ
This one guess is called the forward pass .
θ θ x oppositely : data parallelism copies θ x θ
L L Loss ) is a single number saying how wrong a prediction was. Small L L L ( f ( x ; θ ) , y ) f ( x ; θ ) y
number , not a yes/no
If wrongness were just "right/wrong", we would not know which direction to slide the dials. A continuous score gives us a slope to walk down — which is the whole next section.
This is the symbol beginners fear most. It is just an arrow that points uphill.
∇ θ L
The symbol ∇ "the slope with respect to..." .
The little subscript θ "...with respect to the dials θ
So ∇ θ L one number per dial , telling us: "if you turn this dial up a hair, the loss goes up by this much."
Intuition Picture the gradientImagine standing on a hilly landscape where height = loss . The gradient is the arrow pointing straight uphill (steepest increase). To reduce loss we walk the opposite way — downhill. That is why the update below has a minus sign.
Computing ∇ θ L backward pass (or backpropagation). The parent's "backward pass" box is exactly this. Details live in 3.9-Gradient-descent-optimizers .
Common mistake Why the gradient, not just "try both directions"?With billions of dials, testing each one by hand is impossible. The gradient gives all the slopes at once from a single backward pass — that is why this tool and not brute force.
Intuition Reading the whole line"New dials = old dials, moved a step of size η the entire act of learning . Distributed training changes only where the pieces of this line are computed , never the line itself.
Definition Batch and mini-batch
B batch : a bunch of examples processed together before one update.b = ∣ B ∣ batch size : how many examples are in it. The bars ∣ ⋅ ∣ We do not update on one example at a time (too jittery) nor on all N
Intuition The linearity that makes distribution possibleBecause the batch gradient is just an average , splitting a batch across k identical result to one big machine. This "average of averages" fact is the mathematical seed of data parallelism and of gradient accumulation in the parent note.
average the machines’ gradients k 1 i = 1 ∑ k g i = same number! g one big batch
When k g i all-reduce is the group operation that (1) sums or averages them and (2) hands the same final result back to every machine. "All" = everyone gets it; "reduce" = combine many into one.
Picture: each chef writes their gradient on a card, the cards are combined into one master card, and a copy of the master card is returned to every chef .
Intuition Why "all", why not just one boss machine?If only a boss computed the average, every worker would have to wait for and then re-download it. All-reduce spreads both the adding and the sharing evenly, which is why it is the standard in synchronous data-parallel training. It runs over fast links — see 4.2-GPU-acceleration and 5.1-Training-at-scale .
Definition Timing and count symbols
k how many devices (machines/GPUs) we use.T compute doing math (forward + backward).T comm communicating (the all-reduce of §8).S speedup : how many times faster than a single device.
Gradient of L (backward pass)
All-reduce across k devices
Distributed training overview
Read top-down: data and dials meet in the forward pass, the loss scores it, the gradient says how to fix it, averaging + all-reduce lets many machines agree, the update rule applies the fix, and the speedup relation tells us how much we actually gained. That bundle is the parent topic.
Cover the answer and test yourself. If any line is fuzzy, re-read its section above before touching the parent note.
What does θ data parallelism? The full set of model dials (weights/biases); in data parallelism it is copied to every device.
What does the symbol ∇ θ L Straight uphill — the direction of steepest loss increase ; we step the opposite way.
Why is there a minus sign in θ ← θ − η ∇ θ L Because the gradient points uphill and we want to descend, so we move opposite to it.
What is η The learning rate (step size); too large a step overshoots and can make loss blow up.
Why can a batch gradient be split across machines and re-averaged safely? Because it is an average (a linear operation), so an average of partial averages equals the single big-batch gradient.
In plain words, what does an all-reduce do? Combines every device's gradient into one summed/averaged result and gives that same result back to all devices.
In the speedup formula, which term stops S k The communication time T comm
What is the difference between x y x y
