AdaGrad and RMSprop: Question bank — AI-ML deep dive

A question bank of the misconceptions and boundary cases that AdaGrad and RMSprop invite. Each answer is real reasoning, not a bare yes/no — cover the answer, argue it yourself, then reveal. Parent: 3.2.4 AdaGrad and RMSprop.

Before you start, make sure these three symbols are clear in your head (the whole bank leans on them):

The step size you actually take in direction $i$ is $\frac{η}{(accumulator) + ϵ}$ — that "effective learning rate" is what almost every trap below is really about.

Spot the error

"AdaGrad divides

η

G_{t, i}

, so bigger history means smaller steps."

The scaling divides by

G_{t, i}

, not

G_{t, i}

. Dividing by

G

directly would over-shrink (wrong units:

G

is gradient-squared) and kill learning even faster; the square root keeps the step in gradient-sized units.

"RMSprop's

E [g^{2}]

is an average, so I should divide by the number of steps like a normal mean."

No — the EWMA already carries its own normalization through the

(1 - ρ)

and

ρ

weights, which form a geometric series summing to 1. Dividing again by

t

would double-normalize and shrink the estimate wrongly.

"To make AdaGrad train longer, just increase

η

A bigger

η

delays the death but cannot prevent it: the step is

η / (g t)

, which still

\to 0

for any fixed

η

. The growth of

G_{t}

is the disease; scaling

η

only shifts the timeline, so RMSprop's decaying memory is the real cure.

"For a parameter whose gradient is always exactly 0, AdaGrad and RMSprop will slowly drift it."

g_{t, i} = 0

for all

t

, then the numerator

g_{t, i}

0

, so the whole update term is

0

— the parameter never moves. The accumulator being tiny is irrelevant when the gradient it multiplies is zero.

"RMSprop removes the vanishing/exploding gradient problem."

It mitigates the effect by rescaling each coordinate to a comparable step size, but it does not fix the underlying Vanishing and Exploding Gradients in the backward pass — the gradients themselves can still be tiny or huge; RMSprop just normalizes how far you step given them.

"Adam is a totally different algorithm from RMSprop."

Adam Optimizer is essentially RMSprop plus Momentum plus bias correction — it reuses RMSprop's exact squared-gradient EWMA for the denominator, then adds a first-moment EWMA on top. It's a superset, not a rival.

Why questions

Why do we accumulate squared gradients instead of the gradients themselves?

Raw gradients can be positive or negative and would cancel in a running sum, telling you nothing about typical magnitude. Squaring makes every contribution positive so the accumulator measures how big gradients have been, which is what we want to divide by.

Why does RMSprop need a decay

ρ

at all, instead of a plain running average?

A plain running average

\frac{1}{t} \sum g_{τ}^{2}

still weights ancient gradients as much as fresh ones and its update shrinks like

1/ t

, so it stiffens over time much like AdaGrad. The geometric decay

ρ

keeps memory finite and constant-length, so recent curvature always dominates.

Why is a per-coordinate step size useful when Stochastic Gradient Descent with one

η

works on many problems?

When directions have wildly different curvature — steep in some, flat in others — a single

η

must be small enough not to blow up the steep ones, which then crawls on the flat ones. Per-coordinate scaling lets each direction move at its own safe speed simultaneously.

Why does dividing by

G

tend to equalize step sizes across coordinates?

A coordinate with large gradients builds a large

G

, so dividing shrinks its step; a coordinate with small gradients keeps

G

small and its step large. The net effect pushes very different raw-gradient magnitudes toward comparable actual moves.

Why can't we just use a fixed learning-rate schedule instead of these adaptive methods?

A schedule changes the global rate over time but treats all parameters identically; it cannot notice that parameter A needs shrinking while parameter B needs boosting right now. Adaptivity is per-coordinate and data-driven, which a fixed schedule can't be — though the two are often combined.

Why does AdaGrad still shine on sparse problems (like NLP embeddings) despite its death flaw?

For a rarely-seen feature,

G

stays tiny (few nonzero gradients), so its effective rate remains large exactly when the feature finally appears. The "remember forever" behaviour is a feature for sparse, mostly-convex settings and only a bug for long dense deep-net training.

Edge cases

At the very first step (

t = 1

) with

g_{1, i} \neq = 0

, what is AdaGrad's effective rate?

G_{1, i} = g_{1, i}^{2}

, so the step is

η g_{1, i} / (∣ g_{1, i} ∣ + ϵ) \approx \pm η

— the first gradient sets its own scale, giving a step of size roughly

η

regardless of how big

g_{1, i}

is.

What happens on the first RMSprop step if

E [g^{2}]_{0}

is initialized to 0?

E [g^{2}]_{1} = (1 - ρ) g_{1}^{2}

, which is much smaller than

g_{1}^{2}

, so the denominator

E [g^{2}]_{1}

is under-estimated and the first few steps can be too large. This warm-up bias is exactly what Adam's bias correction fixes.

If two coordinates always have identical gradients, do AdaGrad/RMSprop ever treat them differently?

No — identical gradient histories produce identical accumulators and identical effective rates, so they step in lockstep. Adaptivity only distinguishes coordinates whose gradient magnitudes have differed at some point.

What if

ϵ

is set far too large (say

ϵ = 1

The denominator becomes dominated by

ϵ

, so the effective rate flattens toward

η / (accumulator + 1)

and the per-coordinate adaptation is drowned out — you drift back toward ordinary SGD-like behaviour, losing the whole point.

For a gradient that suddenly spikes then returns to normal, how do the two accumulators react differently?

AdaGrad's

G

absorbs the spike permanently into its sum, so that coordinate stays over-shrunk for the rest of training. RMSprop's EWMA lets the spike decay away in a few steps (roughly

1/ (1 - ρ)

steps of memory), so it recovers a normal step size.

In the limit

t \to \infty

with bounded nonzero gradients, compare the two effective rates.

AdaGrad's

\to 0

because

G_{t}

grows unboundedly, so learning halts. RMSprop's

\to η / \overline{g^{2}}

, a finite nonzero constant set by the recent mean-square gradient, so learning continues indefinitely.

Recall One-sentence summary of every trap here

Almost every misconception collapses once you remember: the accumulator is under a square root, it scales magnitude per coordinate (never sign or direction), and the only real difference between the two methods is grows-forever (AdaGrad, dies) versus decaying-average (RMSprop, survives).

Connections

AdaGrad and RMSprop — the parent this bank stress-tests
Stochastic Gradient Descent — the one-rate baseline several traps contrast against
Exponentially Weighted Moving Averages — the decay math behind RMSprop's survival
Momentum — first-moment averaging, distinct from these second-moment methods
Adam Optimizer — RMSprop + momentum + bias correction (the warm-up trap)
Learning Rate Schedules — global-rate alternative some traps compare to
Vanishing and Exploding Gradients — what these mitigate but don't cure

Question bank — AdaGrad and RMSprop

True or false — justify

Spot the error

Why questions

Edge cases

Connections