AdaGrad and RMSprop: Visual walkthrough — AI-ML deep dive

This page rebuilds the whole idea from a single blank picture: a ball rolling down a valley, one step size that cannot fit two directions, and the exact machinery that fixes it. Every symbol is earned before it appears. Parent: AdaGrad and RMSprop.

Before we touch any formula, let us agree on the picture everything lives in.

Step 1 — What a "gradient" is, as an arrow

WHAT. At our current spot $θ_{t}$ (our position at step $t$ ), the gradient is an arrow that points in the direction of steepest increase of $L$ . We call it $g_{t}$ . Crucially, it is measured right where we are standing — the arrow depends on the current position $θ_{t}$ , so it changes as we move.

WHY an arrow, and why this arrow. To descend, we must know which way is "downhill". The gradient is the one object that packs both direction (where the slope tips) and magnitude (how steep) into a single arrow. We step opposite to it — downhill.

PICTURE. Look at the two arrows below. The tail sits on our current position. In the steep direction the arrow is long; in the flat direction it is short.

Plain Stochastic Gradient Descent then takes this step:

$θ_{t + 1, i} = θ_{t, i} - one step size η slope here at θ_{t} g_{t, i}$

$θ_{t, i}$ ::: our current position on axis $i$ .
$η$ ::: the single learning rate — the same number for every axis.
the minus sign ::: we move against the uphill arrow, i.e. downhill.

Step 2 — Why one $η$ cannot serve two directions

WHAT. Consider a valley that is steep across and gentle along (a long narrow ravine).

WHY it breaks. One $η$ multiplies both the long arrow and the short arrow. Pick $η$ large enough to make progress along the gentle floor, and the steep-across step overshoots — the ball bounces wall to wall. Pick $η$ small enough to be calm across, and along the floor you crawl. There is no single number that is both.

PICTURE. The yellow path zig-zags because the across-step is too big; the along-progress is tiny. This is the disease.

Step 3 — Measuring "how big have the gradients been" with a square

WHAT. We need a single positive number per axis that grows when its gradients are large. We build it by squaring each gradient and adding them up:

$G_{t, i} = \sum_{τ = 1}^{t} g_{τ, i}^{2} = G_{t - 1, i} + g_{t, i}^{2}, G_{0, i} = 0$

$g_{τ, i}^{2}$ ::: the gradient at past step $τ$ , squared → always $\geq 0$ , and big gradients count extra.
$\sum_{τ = 1}^{t}$ ::: add up every step from the first to now.
the right form $G_{t - 1, i} + g_{t, i}^{2}$ ::: same thing written as "yesterday's total plus today's square" — cheap to update.
$G_{0, i} = 0$ ::: we start the accumulator empty (before any gradient has arrived, there is nothing summed yet). This is why the very first divisor is only $g_{1, i}^{2} = ∣ g_{1, i} ∣$ , and why $ϵ$ in Step 4 matters on step 1.

WHY square, not absolute value. Two reasons, both visible in the figure. First, squaring makes it always positive, so it can safely act as a size. Second, squaring punishes large gradients more (a gradient of $2$ contributes $4$ , a gradient of $0.5$ contributes $0.25$ ) — exactly the directions we want to slow down. It is also smooth (differentiable), unlike the sharp corner of $∣ g ∣$ .

PICTURE. Bars show raw $∣ g ∣$ vs $g^{2}$ : the tall gradient's bar shoots up disproportionately after squaring.

Step 4 — Turning that "size" into a step: divide by $G$

WHAT. We now shrink the step of a busy axis by dividing by its accumulated size. AdaGrad's update:

WHY $G$ and not $G$ itself? This is a units argument, and the picture makes it concrete. $G$ carries units of $(gradient)^{2}$ . If we divided $g$ by $G$ , we would over-cancel and the step would collapse far too fast. Taking $G$ brings the divisor back to the same units as $g$ , so the ratio $g / G$ is well-scaled — roughly "how many typical-gradients big is today's gradient". That is exactly the balanced quantity we wanted.

PICTURE. The curve $η / G$ falls off gently; $η / G$ plunges. Gentle is what keeps learning healthy early on.

Step 5 — The fatal flaw: $G$ only grows, so the step dies

WHAT. $G_{t, i}$ is a sum that never decreases — starting from $G_{0, i} = 0$ , every step adds a non-negative $g^{2}$ . So $G_{t, i}$ climbs forever, and the effective rate $\frac{η}{G _{t, i}} \to 0$ .

WHY this is fatal in deep learning. In convex or sparse problems (e.g. NLP embeddings) a decaying rate is fine — you settle near the answer and stop. But deep nets train for hundreds of thousands of steps. Long before the end, the step has shrunk to a micro-step in every direction: the network stops learning. AdaGrad arrests itself.

PICTURE. For a constant gradient $g = 1$ , $G_{t} = t$ , so the effective rate is $η / t$ — a curve visibly sagging toward the floor.

Step 6 — RMSprop: forget the old, remember the recent

WHAT. We keep the "divide by root of squared-gradient size" idea, but we stop summing forever. Instead of a sum we keep a running average that fades old values, called an exponentially weighted moving average (EWMA):

$E [g^{2}]_{t, i} = ρ E [g^{2}]_{t - 1, i} + (1 - ρ) g_{t, i}^{2}, E [g^{2}]_{0, i} = 0$

$E [g^{2}]_{t, i}$ ::: our current estimate of the recent mean-square gradient on axis $i$ .
$ρ$ ::: the decay (memory length), typically $0.9$ . Large $ρ$ = long memory.
$ρ E [g^{2}]_{t - 1, i}$ ::: keep $90%$ of the old estimate — the past fades but is not erased.
$(1 - ρ) g_{t, i}^{2}$ ::: mix in $10%$ of today's fresh square.
$E [g^{2}]_{0, i} = 0$ ::: we also start this estimate empty — the same convention as $G_{0}$ .

WHY this fixes the death. Because old squares are multiplied by $ρ$ again and again ( $ρ, ρ^{2}, ρ^{3}, \dots$ ), their weight decays geometrically to nothing. The estimate no longer piles up without bound — it settles to the recent mean-square. So the divisor stops growing, and the effective rate stays alive.

PICTURE. The weight given to the gradient from $k$ steps back is $(1 - ρ) ρ^{k}$ — a decaying staircase. Old bumps are forgotten.

WHY the $ϵ$ is still earned here. In Step 4 we introduced $ϵ$ to survive $G \to 0$ at start-up. RMSprop's divisor $E [g^{2}]$ can also dip near zero — precisely when an axis's recent gradients are tiny. Without $ϵ$ that would blow the step up to infinity, so the same guard is kept, doing the same job.

Step 7 — RMSprop stays alive: the steady state

WHAT. Feed a constant gradient $g = 1$ into the EWMA (starting from $E [g^{2}]_{0} = 0$ ) and watch where it lands after the early bias fades.

WHY it settles — line by line. "Steady state" means the estimate has stopped changing: the value this step equals the value last step, so we may call both simply $E$ . Substituting that single symbol into the update gives

$this step E = ρ last step (equal now) E + (1 - ρ) g^{2} .$

Why we can write one $E$ on both sides: that is exactly what "settled / no longer changing" means. Now solve for $E$ :

$E - ρE = (1 - ρ) g^{2} (move the ρE term to the left — collect all E ’s together)$ $(1 - ρ) E = (1 - ρ) g^{2} (factor out E on the left)$ $E = g^{2} (divide both sides by (1 - ρ), valid since ρ < 1 so 1 - ρ \neq = 0).$

So the settled estimate is exactly the true mean-square $g^{2}$ — the fading average lands on the honest recent size, no bigger, no smaller.

WHY it matters. With $g = 1$ , $E \to 1$ , so the effective step $\to η / 1 = η$ — constant forever, never zero.

PICTURE. AdaGrad's curve ( $η / t$ ) sinks; RMSprop's curve starts a touch high (the early bias) then flattens onto a stable plateau. Same input, two fates.

The one-picture summary

Everything in one frame: SGD's one step size fails in the ravine → measure recent gradient size per axis → divide $η$ by its square root. AdaGrad measures with a growing sum (sinks to zero); RMSprop measures with a fading average (holds a healthy plateau). RMSprop + a momentum term is exactly the Adam Optimizer.

Recall Feynman retelling of the whole walkthrough

You are rolling a ball into a bowl in the dark; the height of the bowl is the loss $L$ (how wrong the network is), and all you feel is the slope under the ball (the gradient arrow), measured right where the ball sits. If the bowl is a narrow ravine — steep across, gentle along — one step size can't win: big enough to move along means bouncing off the walls; small enough to be calm means crawling. So we give each direction its own step size. To decide it, we square each slope we've felt (squares are always positive and punish big slopes more), start a tally at zero, and use the square root of the tally as a "typical size" to divide by — big divisor means small step, exactly for the busy directions. AdaGrad divides by the root of the sum of all squares — but a sum only grows, so eventually every step becomes a micro-step and the ball stops. RMSprop divides by the root of a fading average instead: it keeps mostly the recent slopes and forgets old ones. It too starts its tally at zero, so its very first steps are a little too big, but that bias fades fast — and after that the divisor settles instead of exploding, so the ball keeps rolling at a steady pace all the way down.

Recall

AdaGrad divisor vs RMSprop divisor? ::: AdaGrad: $sum of all past g^{2}$ (grows, dies). RMSprop: $EWMA of g^{2}$ (bounded, survives). Why $G$ not $G$ ? ::: Units — $G$ matches the units of $g$ , giving a well-scaled step; $G$ over-shrinks. What do the accumulators start at? ::: Both $G_{0, i} = 0$ and $E [g^{2}]_{0, i} = 0$ (empty). Steady-state RMSprop step for constant $g$ ? ::: $E [g^{2}] \to g^{2}$ , so effective step $\to η$ .

Connections

Stochastic Gradient Descent — the one- $η$ baseline we improved.
Momentum — averages the gradient itself (first moment); here we average its square (second moment).
Adam Optimizer — RMSprop + momentum + bias correction (fixes the start-at-zero bias of Step 6).
Exponentially Weighted Moving Averages — the fading-memory engine of Step 6.
Learning Rate Schedules — hand-designed decay, an alternative to adaptivity.
Vanishing and Exploding Gradients — per-axis scaling tames wildly different gradient sizes.