Intuition The ONE core idea of this whole topicAdam is a rule for updating numbers (the network's weights) a little at a time so a loss shrinks — and its trick is to give each weight its own step size by remembering two things about that weight's recent gradients: their average direction and their typical size . Everything below is just the vocabulary you need before that sentence makes full sense.
This page assumes you have seen nothing . We name every letter, draw the picture it stands for, and say why the topic can't live without it. Read top to bottom — each item uses only the ones above it.
Before any Greek letters, picture the whole job.
θ parameter is a single tunable number inside the network — a "knob" we are allowed to turn. We write one such knob as θ theta ). A real network has millions of them; think of θ any one knob , and remember the same rule runs on all of them independently.
Picture: a slider you can drag left/right. Its position is a number.
L loss L how wrong the network currently is. Big L L Picture (see figure): a landscape where height = L downhill .
g ∇ θ L gradient g "if I nudge this knob up a tiny bit, does the loss go up or down, and how steeply?" The notation ∇ θ L L θ
Sign of g opposite way.Size ∣ g ∣
Picture (see figure): an arrow on the loss landscape pointing straight uphill. We move against it.
Intuition Why a slope and not the loss value itself?Knowing the loss is 5 tells you nothing about which way to move. Knowing the slope tells you the direction and steepness — exactly the two facts a step needs. That is why every optimizer is built on g L
The subscript t g t θ t "at time step t — step 1, step 2, step 3… One minibatch of data = one t
η η eta ) is how big a step we take each time. Small η η Picture: the length of each footstep as you walk downhill.
The simplest possible rule, Stochastic Gradient Descent , is:
θ ← θ − η g t . ← "becomes" : the new knob value is the old one minus a step of size η down the hill, not up.
Intuition Why SGD is not enough (the gap Adam fills)One global η every knob. But some knobs sit on cliffs (huge g g give each knob its own effective step size.
Adam never trusts a single noisy gradient. It keeps a running smoothed value . The tool for that is the exponential moving average.
Definition Exponential moving average (EMA) and decayβ x 1 , x 2 , x 3 , … EMA is built one step at a time:
m t = β m t − 1 + ( 1 − β ) x t , m 0 = 0. decay β beta ), a number between 0 1 memory length :
β 1 β 0
Picture (see figure): noisy dots (the raw x t fades geometrically — that's the "exponential" in the name.
Why is this the exponential moving average? Unrolling the recursion, the value from k β k β k Exponential Moving Average for the full unrolling; Adam uses it twice.
Intuition Why average at all?Each minibatch gradient g t noisy sample — a slightly wrong estimate of the true slope (this noise is exactly the subject of Bias-Variance in Gradient Estimation ). Averaging many samples cancels the random wobble and reveals the trend. See Momentum and Nesterov Acceleration for how this same average also builds "momentum" down a valley.
The parent note calls β t t β t β < 1 0 t bias correction .
Adam runs the EMA machine on two different streams .
m t x t = g t m t = β 1 m t − 1 + ( 1 − β 1 ) g t . m t average gradient — mostly which way to go. Its decay is called β 1 0.9 Picture: a smoothed arrow that ignores single-step jitter.
v t square of the gradient, x t = g t 2 v t = β 2 v t − 1 + ( 1 − β 2 ) g t 2 . g t 2 v t typical size of the gradient in this coordinate. Its decay is β 2 0.999 β 1 RMSProp .
Picture: a smoothed "how bumpy is it here" meter.
Common mistake "Why square the gradient — why not use∣ g t ∣ Why it feels right: the absolute value also measures size. The fix: squaring makes the math of expectations clean (v t E [ g 2 ] second moment ), and v t
The little hats — m ^ t v ^ t bias-corrected versions, m t / ( 1 − β 1 t ) v t / ( 1 − β 2 t ) m 0 = v 0 = 0 ( 1 − β t ) a hat means "fixed for the cold start."
x square root v ^ t v ^ t v ^ t m ^ t
ϵ ϵ epsilon , default 1 0 − 8 v ^ t + ϵ v ^ t 0 and it caps how large the effective step can grow.
Picture: a small pebble under the seesaw so it can never touch the ground.
λ λ lambda ) controls how hard we shrink every weight toward zero each step, the − η λ θ λ Weight Decay and L2 Regularization .
Picture: a gentle spring always tugging each slider back toward 0
The figure shows the ratio m ^ t / v ^ t ≈ ± 1 ≪ 0 the reason Adam adapts per-parameter.
second moment v magnitude
Each arrow means "you need the left box before the right one makes sense." Notice how everything funnels through the gradient g — that single arrow is the whole reason optimizers work.
Cover the right side and test yourself.
What does the knob θ One tunable number (weight) inside the network; the same update runs on every one.
What single number is L The loss — a badness score we try to make small.
What two facts does the gradient g Its sign (which way is uphill) and its size ∣ g ∣
Why do we step against g g
What does the learning rate η The length of each step.
Write the EMA recursion from memory. m t = β m t − 1 + ( 1 − β ) x t m 0 = 0
What does β 1 Makes it long / very smooth / slow to react.
What stream feeds m t v t m t g t v t g t 2
What does a "hat" (m ^ , v ^ Bias-corrected — un-shrunk to fix the cold start from 0
Why divide by v ^ t v ^ t The square root restores original gradient units so numerator and denominator match.
What two jobs does ϵ Prevents divide-by-zero and caps the maximum effective step.
What does λ The strength of uniform weight decay pulling each weight toward zero.