Intuition The one core ideaTraining a network means finding the settings (the numbers inside it) that make its mistakes as small as possible — and the way we search is to repeatedly ask "which tiny nudge to the settings makes the mistake smaller?" and take that nudge. Mini-batch gradient descent just answers that question using a small random pile of examples at a time instead of all of them, because a small pile gives a good-enough answer far faster.
This page assumes nothing . Every letter, arrow, and squiggle in the parent note is built here from the ground up, each one leaning on the one before it. Read top to bottom.
Before any math, a picture of what we are even adjusting.
A neural network is a machine: you feed in some input (say the pixels of a photo) and it produces an output (say "cat" or "dog"). Between input and output sit a large pile of numbers — the knobs. Turn the knobs and the machine's behaviour changes. Training = turning the knobs until the machine is usually right.
Definition The symbols for the machine
x input example (one photo, one row of data). Picture: the thing you feed in.y correct answer (the "label") that goes with that x f model , the machine itself: it turns an input into a guess.θ bundle of all the knob settings . Picture: one giant list of every adjustable number inside f f θ ( x ) f set up with knobs θ , applied to input x guess for that input.
Why the topic needs θ everything in training is about changing θ the variable we optimise. We write f θ
We can only improve if we can measure how wrong we are . That measurement is called the loss .
Definition The loss of one exampleℓ ( f θ ( x i ) , y i ) i f θ ( x i ) y i one number — small if the guess is good, large if it's bad.
ℓ loss function , a wrongness-meter.The subscript i index , just a counter naming which example ("the i
Intuition Why a number and not "right/wrong"?A yes/no verdict gives us no direction to improve. A number that grows smoothly as the guess gets better tells us not just that we are wrong but how wrong — and later, which way to turn the knobs to be less wrong. That smoothness is the whole reason loss is a number.
We often shorten ℓ ( f θ ( x i ) , y i ) ℓ i i
We don't care about one example; we care about all of them. That needs a way to add many things.
Definition The summation symbol∑ i = 1 N ℓ i = ℓ 1 + ℓ 2 + ℓ 3 + ⋯ + ℓ N
∑ "add up" .i = 1 N N N total number of training examples (the size of your whole dataset).
Picture: a conveyor belt carrying N ∑
To turn a total into an average , divide by how many there are:
Why the topic needs J J ( θ ) down . Its height at position θ J
Now the key move. J θ θ location and J altitude there. Training is descending this landscape. To descend, we need to know which way is downhill . That direction-finder is the gradient.
Intuition What a gradientis , in plain words
Stand on a hillside blindfolded. Feel the slope under your feet. The gradient is the arrow pointing in the direction of steepest uphill , and its length says how steep . To go down , walk the opposite way. That is the entire trick of gradient descent.
Definition The gradient symbol∇ ∇ θ J ( θ )
∇ "the vector of slopes of J The subscript θ with respect to the knobs θ J
The result is itself a list of numbers , one per knob, pointing in the steepest-uphill direction.
Why we need it: it is the only thing that tells us which way to nudge θ
Definition Per-example gradientg i = ∇ θ ℓ i example i . The full gradient is the average of all these arrows:
∇ θ J = N 1 ∑ i = 1 N g i
Knowing the downhill direction, we take a step. How big a step? That is the learning rate.
Definition Learning rate and the updateθ t + 1 = θ t − η g
η learning rate , a small positive number scaling the step size. Picture: your stride length.g The minus sign — we step against the uphill arrow, i.e. downhill.
t time counter : θ t θ t + 1
Intuition Why not just jump to the bottom?The gradient only tells you the slope right here ; it doesn't know where the valley bottom is. So you take a small step, look again, step again. η η Learning Rate Scheduling .
Mini-batch GD estimates the true average gradient from a random pile . To reason about "estimate" we need three ideas from probability.
Definition The random-sampling vocabulary
B mini-batch : a small random subset of the example indices { 1 , … , N } m batch size , how many examples are in B m ≪ N m much smaller than N E [ ⋅ ] expectation ("expected value"), the long-run average if you repeated the random draw forever. Picture: the value the dice average out to.Var ( ⋅ ) variance , how much the random thing spreads around its average. Picture: wide scatter = big variance, tight cluster = small variance."std" = standard deviation = Var
Intuition Why these three matter for the topic
Expectation lets us say the batch gradient is right on average (unbiased): E [ g B ] = ∇ J Variance measures the noise in one batch's guess — and the parent's headline result is that this noise shrinks like σ / m σ
≪ < Why it feels right: both are "less than."
The fix: ≪ much less than — orders of magnitude (m = 128 N = 1 0 7 "the batch is a tiny sample of a huge dataset," which is exactly why sampling saves so much work.
One epoch = one full pass through all N m = ⌈ N / m ⌉ ⌈ ⋅ ⌉ round up (the last, smaller batch still counts). Picture: reading a whole book once = one epoch; each page-turn = one batch update.
Sum and average gives objective J
Gradient nabla J points downhill
Update rule with learning rate eta
Random sample batch B of size m
Mini batch gradient descent
Read it as: knobs and loss build the objective J J batch estimate ; both streams meet at mini-batch GD.
Hide the answers, say each out loud, then reveal.
What does θ The bundle of all adjustable knob-settings inside the model; picture a giant list of numbers we turn during training.
What does f θ ( x ) The model f θ x x
What is ℓ i The loss (wrongness) on example i how wrong and hence which way to improve, which yes/no cannot.
Expand ∑ i = 1 N ℓ i Add up the losses of examples 1 through N into one bucket.
Why divide the total loss by N J To make it an average, so the value doesn't secretly grow just because the dataset is bigger.
What does the gradient ∇ θ J Toward steepest uphill of the loss; we step the opposite (downhill) direction.
In θ t + 1 = θ t − η g η η
What is a mini-batch B m ≪ N A small random handful of examples; m ≪ N
What do E [ ⋅ ] Var ( ⋅ ) Expectation = long-run average of a random quantity; variance = how much it spreads around that average.
What is one epoch, and how many steps does it contain? One full pass over all N ⌈ N / m ⌉
