Before you can read the parent note Learning rate effects on convergence, you must be able to read every squiggle it uses without hesitation. This page builds each one from nothing, in an order where every symbol is earned by the one before it. Nothing here assumes you have seen calculus notation before.
In the parent note you see θt with a little subscript t. That t just labels which step in time we are at: θ0 is the starting knob setting, θ1 is after one adjustment, θ2 after two, and so on. The subscript is a counter, nothing more.
The notation L(θ) is read "L of theta": you hand it a knob setting θ, it hands you back a wrongness number. The parentheses mean "feed this in", exactly like a vending machine slot.
Suppose you are standing blindfolded on the loss landscape. You want to go downhill but you cannot see the whole valley — you can only feel the ground right under your feet. What can you feel? The slope: which way is up, which way is down, and how steep it is.
Now, most models have many knobs at once. The slope in each knob direction gets bundled into one arrow. That bundle is the gradient.
The little subscript θ in ∇θ just says "take the slopes with respect to the knobs θ" — it names which sliders we're feeling under our feet. The superscript T you sometimes see (∇LTΔθ) is the "transpose" — a bookkeeping mark that lets two arrows be multiplied into a single number (a dot product). You do not need it to grasp the idea of a step. See Gradient Descent for how the arrow drives every update.
Now we can finally define the star of the topic honestly, because every piece it depends on exists.
η is called a hyperparameter: a number you pick before training, as opposed to θ which the training learns. Later notes replace the single fixed η with smarter schemes — see Learning Rate Schedules, Momentum, and Adam Optimizer.
Every arrow means "you need the left box to understand the right box." Notice the two streams — the step-building stream (top) and the curvature stream (bottom) — meet at the stability rule, which is the heart of the parent topic.
Cover the right side and answer each. If any answer surprises you, reread that section.
What does θ represent, in plain words?
The setting(s) / knob(s) we are allowed to adjust to improve the model.
What does the subscript t in θt mean?
A time/step counter — which iteration we are on.
What does L(θ) give you when you feed in a θ?
A single number measuring how wrong that setting is (loss / height on the landscape).
Which direction does ∇θL point?
Steepest uphill — so we move opposite it to go down.
Why is there a minus sign in θt+1=θt−η∇L?
To turn the uphill gradient into a downhill move.
What does Δθ mean?
The change in the knob during one update, θt+1−θt.
In one phrase, what is η?
The step length — how far we move per update.
Direction vs distance: which does the gradient give, and which does η give?
Gradient gives direction; η gives distance.
What does a large smoothness constant L force you to do with η?
Make η smaller (sharper valleys need shorter steps); rule is η<2/L.
For the bowl L(w)=w2, what is the curvature λ and the stability limit on η?
λ=2, so η<2/λ=1.
What is the sole purpose of ϵ in the adaptive-rate denominator?
To prevent division by zero (tiny safety constant).
Recall Quick self-check before moving on
If you can read θt+1=θt−η∇θL(θt) out loud as a full sentence and point to the picture for each symbol, you are ready for the parent note.
Read the update rule as an English sentence. ::: Next setting = current setting minus (step length) times (steepest-uphill arrow) — i.e. take a step of size η downhill.