AdaGrad and RMSprop

Adaptive learning-rate optimizers that give each parameter its own step size, scaled by the history of its gradients.

The core problem (WHY these exist)

WHAT we want: a rule that automatically shrinks the step for parameters with big/frequent gradients and keeps large steps for parameters with small/rare gradients.

HOW: track the accumulated squared gradient per parameter, and divide the learning rate by its square root.

AdaGrad — derivation from scratch

Let $g_t = \nabla_\theta L$ at step $t$, and consider one coordinate $i$ so $g_{t,i}$ is a scalar.

Idea: the "typical size" of the gradient in coordinate $i$ is captured by summing its squares. Big gradients ⇒ big sum ⇒ we want a smaller effective step. The natural way to convert a "size" into a divisor is to divide by it. Squared to make it always positive and to weight large gradients more.

Define the running sum of squares: $G_{t,i} = \sum_{\tau=1}^{t} g_{\tau,i}^2 = G_{t-1,i} + g_{t,i}^2$

Then the update is:

Why $\sqrt{G}$ and not $G$? Units. $G$ has units of (gradient)$^2$; dividing $g$ by $\sqrt{G}$ makes the ratio dimensionless-ish and gives a well-behaved, bounded step. Dividing by $G$ would over-shrink.

The fatal flaw of AdaGrad

Because $G_{t,i}$ is a sum that only grows, $\sqrt{G_{t,i}}\to\infty$, so the effective learning rate $\to 0$. In deep learning training runs for many steps, AdaGrad dies — it stops learning too early. Great for convex / sparse problems (NLP embeddings), bad for long deep-net training.

RMSprop — the fix

WHY: we don't want the entire history to accumulate forever; we want a recent estimate of gradient magnitude. Old gradients should be forgotten.

HOW: replace the growing sum with an exponentially weighted moving average (EWMA) of squared gradients. This keeps a decaying memory instead of an ever-growing sum.

Define, with decay $\rho$ (typically $0.9$): $E[g^2]_{t,i} = \rho\, E[g^2]_{t-1,i} + (1-\rho)\, g_{t,i}^2$

This is a weighted average: recent squares count fully, old ones fade geometrically. Now it doesn't blow up — it settles to the recent mean-square gradient.

Because $E[g^2]$ stays a bounded average (not a sum), the effective learning rate stops decaying to zero — training continues.

Worked example 1 — AdaGrad on two coordinates

Let $\eta=0.1$, $\epsilon=0$. Coordinate A always has $g=1$; coordinate B always has $g=0.1$.

Step 1:

• $G_A = 1^2 = 1$, step $=\frac{0.1}{\sqrt1}\cdot1 = 0.1$. Why? First gradient sets the scale.
• $G_B = 0.1^2 = 0.01$, step $=\frac{0.1}{\sqrt{0.01}}\cdot0.1 = \frac{0.1}{0.1}\cdot0.1 = 0.1$.

Why interesting: even though B's raw gradient is 10× smaller, AdaGrad rescaled its step up to match A. That's the equalizing effect.

Step 2:

• $G_A = 2$, step $=\frac{0.1}{\sqrt2}\cdot1 \approx 0.0707$. Why? History accumulated, step shrank.

Worked example 2 — RMSprop stays alive

$\eta=0.1$, $\rho=0.9$, $\epsilon=0$, constant $g=1$.

$E[g^2]$ converges: at steady state $E = 0.9E + 0.1(1) \Rightarrow E = 1$. So step $=\frac{0.1}{\sqrt1}\cdot1 = 0.1$ forever. Why this step? The EWMA saturates at the true mean-square (1), giving a constant effective rate — no death. Contrast AdaGrad where $G_t=t\to\infty$ and step $\to 0$.

Flashcards

Connections

• Stochastic Gradient Descent — the baseline these improve on
• Momentum — smooths gradients (first moment) vs these which scale by variance (second moment)
• Adam Optimizer — literally = RMSprop + momentum + bias correction
• Exponentially Weighted Moving Averages — the math engine behind RMSprop
• Learning Rate Schedules — alternative/complement to adaptivity
• Vanishing and Exploding Gradients — why per-coordinate scaling helps

Concept Map

Hinglish (regional understanding)