Intuition The one core ideaTraining a network means repeatedly nudging its numbers downhill on a "badness" surface, and the learning rate is simply how big each nudge is . A learning rate schedule is the plan for shrinking those nudges over time — big early to travel fast, tiny later to settle gently — and everything on the parent page is built from that single picture.
This page assumes nothing . If the parent note used a symbol, we define it here from scratch: plain words → the picture it stands for → why the topic needs it. Read top to bottom; each block only uses ideas already built above.
Intuition The valley pictureA trained network is just a big list of numbers (called parameters ). For any choice of those numbers, we can measure how wrong the network is on our data — one single "badness" number. If you imagine plotting badness against the parameters, you get a bumpy surface with valleys . Training = rolling a ball to the bottom of a valley.
Look at the surface above. The height is badness. The ball is our current network. We want to reach the lowest point. Every symbol below is a piece of this one picture. See Loss Landscapes and Minima for the surface itself.
θ θ whole collection of numbers inside the network — every weight and bias — bundled together into one object. Think of it as the coordinates of the ball on the valley floor-map.
Plain words: "the current setting of all the dials in the network."The picture: a single dot on the ground-map of the valley. Move the dot → you've changed the network.Why the topic needs it: the schedule's whole job is to control how far the dot moves each step , so we must first have a name for the dot.
θ Why it feels right: we write it with one letter.
Why it's wrong: it's usually millions of numbers at once. We use one letter as shorthand for the entire list, exactly like calling a whole shopping list "the list."
L ( θ ) L θ one number = how wrong the network is with those settings. "Loss" is just a technical word for badness .
The picture: the height of the surface directly above the ball's position θ Why we need it: without a height to descend, "downhill" has no meaning. The loss defines which way is down.
L ( θ ) = one badness number for the setting θ
Lower L minimum (see Loss Landscapes and Minima ).
Here a genuine math tool enters. We must justify it.
gradient and not just L L which way is downhill. We need something that points the steepest uphill direction at your feet — then we walk the opposite way. That "steepest-uphill arrow" is exactly the gradient .
∇ θ L gradient (symbol ∇ arrow that, at the ball's location, points in the direction where the loss L increases fastest . Its length says how steep the slope is.
Plain words: "the steepest-uphill arrow at your current spot."The picture: an arrow lying flat on the ground-map, pointing toward higher ground.Why the topic needs it: to go down we step in the opposite direction, − ∇ θ L
Common mistake "The gradient points downhill."Why it feels right: we're descending, so surely it points where we go.
Why it's wrong: by definition it points uphill (steepest increase). We deliberately negate it. Miss the minus and the ball climbs out of the valley.
The subscript θ ∇ θ θ
η η positive number that controls how far the ball moves along the downhill direction each step. Big η η
The picture: the length of each footstep the ball takes downhill.Why we need it: the gradient only gives a direction and a raw steepness; η distance moved . Everything on the parent page is about choosing η
Now we can read the master update, one symbol at a time:
t t how many updates we've done : t = 0 t = 1
The picture: a frame number in the movie of the ball rolling down.Why the topic needs it: a schedule means η changes as t — written η ( t )
step and epoch .
One step = one parameter update (one mini-batch). One epoch = one full pass over all the data = many steps. They differ by (#batches per epoch). Mixing the units makes η t Batch Size and Learning Rate scaling .
η ( t ) η ( t ) rule/curve : give it the step number t
The picture: a curve — time on the bottom axis, step size going up. The ball's stride follows this curve as it descends.Why: the parent's whole thesis — big steps early, small steps late — is literally "make η ( t )
The parent's derivation of why LR must decay needs three probability ideas. Build them now.
Intuition Why randomness appears at allWe don't compute the loss on all data at once — too slow. We use a small random mini-batch , so the gradient we get is the true one plus random wobble . That wobble is why the ball never sits perfectly still. This is the "stochastic" in Stochastic Gradient Descent .
ξ t ξ t random error added to the gradient at step t zero-mean : on average it's nothing, but on any single step it jitters the arrow.
Var ( ⋅ ) σ 2 Variance measures how spread out a random quantity is — how wildly it jitters. σ 2 ξ t
The picture: a fuzzy cloud of possible ball-positions instead of one exact dot. Big variance = wide cloud.Why the topic needs it: the parent shows the ball settles into a buzzing cloud of width V ∝ η η Robbins-Monro Stochastic Approximation .
a L ( θ ) = 2 1 a θ 2 a how steeply the walls rise near the bottom. Big a a
The picture: the tightness of the U-shape. A tight U bounces the ball back hard.Why: it appears in the noise-floor formula and controls whether a given η
The parent's schedule formulas use three notations. Define each and say why that tool .
⌊ x ⌋ ⌊ x ⌋ round down to the nearest whole number . ⌊ 2.9 ⌋ = 2 ⌊ 65/30 ⌋ = ⌊ 2.16 ⌋ = 2 Why this tool: step decay drops η counts how many drops have happened , ignoring the fraction in between.
e − λ t e ≈ 2.718 e − λ t smoothly halves at a steady rate as t λ decay rate — bigger λ Why this tool: it gives a smooth version of step decay with no sudden jumps; its "half-life" is t 1/2 = ln 2/ λ
cos ( ⋅ ) cos + 1 − 1 flat at both ends .
Why this tool: we want a schedule that stays high early , flat/gentle at both ends , and drops fastest in the middle — exactly cosine's shape. That flat-at-the-ends behaviour is the whole point of cosine annealing (and of Warm Restarts (SGDR) ).
Common mistake "Cosine annealing decays in a straight line."Why it feels right: the middle of the arc looks straight.
Why it's wrong: near t = 0 t = T flat , so η
Read top-down: parameters + loss give a gradient; gradient + learning rate give the update; the step counter turns a fixed η η ( t ) proves the schedule must decay; the floor/exp/cos tools build the actual curves. Everything converges on the parent topic. For choosing the numbers themselves, see Hyperparameter Tuning and Adam and Adaptive Optimizers .
Self-test: cover the right side and answer aloud.
What does θ The whole collection of network numbers — the ball's position on the valley map.
What single number does L ( θ ) How wrong (bad) the network is for setting θ
Which direction does the gradient ∇ θ L Steepest uphill ; we step the opposite way to go down.
Why is there a minus sign in θ t + 1 = θ t − η ∇ L To turn the uphill gradient into a downhill move.
In words, what is the learning rate η The size/length of each step the ball takes downhill.
What does t t
What is a schedule η ( t ) A rule/curve giving the step size to use at each step t
What does variance / σ 2 How much the mini-batch gradient noise jitters — the width of the ball's buzzing cloud.
Why does V ∝ η The residual jitter shrinks only as η → 0 η
What does ⌊ x ⌋ Rounds down; it counts how many whole drops have occurred.
Why cosine and not a straight line for annealing? Cosine is flat at both ends (high-then-gentle) and drops fastest mid-way — ideal travel-then-settle shape.
