AdaGrad and RMSprop: Exercises — AI-ML deep dive

Work each problem on paper first, then open the solution. Problems climb from "just recall the rule" to "design a new optimizer." Every number here is machine-checked.

This page is a workout for the parent note. If any symbol feels unfamiliar, re-read the parent first. Quick key, so you never hit an undefined symbol:

Definition Symbol key (everything used below)

$θ_{t, i}$ (theta) — the parameter (a single weight) at coordinate $i$ and step $t$ . This is the number we are actually adjusting; each update nudges it toward a lower loss.
$g_{t, i}$ — the gradient of the loss with respect to parameter $θ_{i}$ at step $t$ . A single number telling you "which way and how hard to push parameter $i$ ."
$η$ (eta) — the base learning rate, one global knob for overall step size.
$ϵ$ (epsilon) — a tiny number ( $\sim 1 0^{- 8}$ ) added under the square root so we never divide by zero.
$G_{t, i}$ — AdaGrad's running sum of past squared gradients for coordinate $i$ .
$E [g^{2}]_{t, i}$ — RMSprop's exponentially weighted moving average of squared gradients (see Exponentially Weighted Moving Averages). In Exercise 3.3 we abbreviate this same quantity as $E_{t}$ when the coordinate $i$ is fixed.
$ρ$ (rho) — RMSprop's decay, how long the average remembers (typically $0.9$ ).
Effective learning rate — the number actually multiplying $g$ after the division: $\frac{η}{accumulator + ϵ}$ .

Throughout, one AdaGrad step is $θ_{t + 1, i} = θ_{t, i} - effective rate \frac{η}{G _{t, i} + ϵ} g_{t, i}, G_{t, i} = G_{t - 1, i} + g_{t, i}^{2},$ and one RMSprop step is $θ_{t + 1, i} = θ_{t, i} - \frac{η}{E [ g ^{2} ] _{t, i} + ϵ} g_{t, i}, E [g^{2}]_{t, i} = ρ E [g^{2}]_{t - 1, i} + (1 - ρ) g_{t, i}^{2} .$

Two figures accompany the analysis problems. Figure 1 (below) plots the effective learning rate over time — the coral curve is AdaGrad, the mint curve is RMSprop; you will reproduce it in Exercises 3.1 and 3.2.

Figure 2 (below) shows RMSprop's automatic braking when a gradient spike arrives; you will compute its numbers in Exercise 4.1.

Level 1 — Recognition

Exercise 1.1

Which quantity does AdaGrad accumulate: the sum of gradients, the sum of squared gradients, or the sum of absolute gradients? State the recurrence.

Recall Solution

AdaGrad accumulates the sum of squared gradients: $G_{t, i} = G_{t - 1, i} + g_{t, i}^{2} .$ Squaring makes every term positive (so the accumulator can only grow) and weights large gradients more heavily than small ones. The later $\cdot$ restores the original gradient's units.

Exercise 1.2

In RMSprop with $ρ = 0.9$ , what fraction of the new accumulator comes from the current squared gradient, and what fraction from history?

Recall Solution

The recurrence is $E [g^{2}]_{t} = ρE [g^{2}]_{t - 1} + (1 - ρ) g_{t}^{2}$ . With $ρ = 0.9$ :

Fraction from history: $ρ = 0.9$ (i.e. $90%$ ).
Fraction from the current squared gradient: $1 - ρ = 0.1$ (i.e. $10%$ ).

Exercise 1.3

True or false: in AdaGrad the effective learning rate can increase from one step to the next. Justify.

Recall Solution

False. $G_{t, i} = G_{t - 1, i} + g_{t, i}^{2}$ and $g_{t, i}^{2} \geq 0$ , so $G$ never shrinks. Since $\frac{η}{G _{t, i} + ϵ}$ has $G$ in the denominator, it can only stay the same (if $g = 0$ ) or decrease. It never increases. This monotone decrease is exactly AdaGrad's eventual death.

Level 2 — Application

Exercise 2.1

AdaGrad, $η = 0.5$ , $ϵ = 0$ . A coordinate sees gradients $g_{1} = 3, g_{2} = 4$ . Compute $G_{1}$ , $G_{2}$ , and the size of both update steps.

Recall Solution

Step 1: $G_{1} = 3^{2} = 9$ . Step size $= \frac{0.5}{9} \cdot 3 = \frac{0.5}{3} \cdot 3 = 0.5$ . Step 2: $G_{2} = 9 + 4^{2} = 25$ . Step size $= \frac{0.5}{25} \cdot 4 = \frac{0.5}{5} \cdot 4 = 0.4$ . Notice the raw gradient grew (3 → 4) but the step shrank (0.5 → 0.4): the growing accumulator wins.

Exercise 2.2

RMSprop, $ρ = 0.9$ , starting $E [g^{2}]_{0} = 0$ . Gradients $g_{1} = 2, g_{2} = 2$ . Compute $E [g^{2}]_{1}$ and $E [g^{2}]_{2}$ .

Recall Solution

$E [g^{2}]_{1} = 0.9 (0) + 0.1 (2^{2}) = 0.1 \cdot 4 = 0.4$ . $E [g^{2}]_{2} = 0.9 (0.4) + 0.1 (4) = 0.36 + 0.4 = 0.76$ . It is climbing toward the true mean-square $g^{2} = 4$ but slowly — the average lags because it only takes $10%$ of each new value. (This lag near $t = 0$ is what Adam's bias correction fixes.)

Exercise 2.3

AdaGrad, $η = 0.1$ , $ϵ = 0$ . Two coordinates: A has constant $g = 1$ ; B has constant $g = 0.01$ . After step 1, what is each effective learning rate?

Recall Solution

A: $G_{A} = 1$ , effective rate $= \frac{0.1}{1} = 0.1$ . B: $G_{B} = 0.0 1^{2} = 1 0^{- 4}$ , effective rate $= \frac{0.1}{1 0 ^{- 4}} = \frac{0.1}{0.01} = 10$ . B's tiny gradient earns a $100 \times$ larger effective rate — the equalizing behaviour that helps rare/sparse features move. (Related: Stochastic Gradient Descent would give both the same $0.1$ .)

Level 3 — Analysis

Exercise 3.1

For AdaGrad with a constant gradient $g = c$ and $ϵ = 0$ , show the effective learning rate at step $t$ is $\frac{η}{c t}$ , and confirm it $\to 0$ .

Recall Solution

With constant $g = c$ : $G_{t} = \sum_{τ = 1}^{t} c^{2} = t c^{2}$ . Then $eff_{t} = \frac{η}{t c ^{2}} = \frac{η}{c t} .$ As $t \to \infty$ , $t \to \infty$ , so $eff_{t} \to 0$ . This is AdaGrad's death: after $10, 000$ steps the rate is $\frac{1}{100}$ of its step-1 value. See the coral curve in Figure 1.

Exercise 3.2

For RMSprop with constant $g = c$ , find the steady-state value of $E [g^{2}]$ and the steady-state effective rate. Compare to 3.1.

Recall Solution

At steady state $E [g^{2}]$ stops changing, call it $E^{⋆}$ : $E^{⋆} = ρ E^{⋆} + (1 - ρ) c^{2} \Rightarrow E^{⋆} (1 - ρ) = (1 - ρ) c^{2} \Rightarrow E^{⋆} = c^{2} .$ The effective rate settles to $\frac{η}{c ^{2}} = \frac{η}{c}$ — a constant, independent of $t$ . Contrast 3.1 where AdaGrad's rate $\frac{η}{c t} \to 0$ . RMSprop stays alive because it averages (bounded) rather than sums (unbounded). This is the mint curve in Figure 1.

Exercise 3.3

Expand the RMSprop recurrence (starting $E [g^{2}]_{0} = 0$ ) to show $E [g^{2}]_{t} = (1 - ρ) \sum_{k = 1}^{t} ρ^{t - k} g_{k}^{2}$ . What does this tell you about how far back RMSprop "remembers"?

Recall Solution

Write $E_{t}$ as shorthand for $E [g^{2}]_{t}$ (same quantity, fixed coordinate). Unroll one step at a time: $E_{t} = (1 - ρ) g_{t}^{2} + ρ E_{t - 1} = (1 - ρ) g_{t}^{2} + ρ (1 - ρ) g_{t - 1}^{2} + ρ^{2} E_{t - 2} = \dots$ $E_{t} = (1 - ρ) \sum_{k = 1}^{t} ρ^{t - k} g_{k}^{2} .$ A gradient from $n$ steps ago carries weight $ρ^{n}$ . With $ρ = 0.9$ , weight $0. 9^{n}$ halves roughly every $ln 2/ ln (1/0.9) \approx 6.6$ steps — so RMSprop's memory is only a handful of steps deep. Old bumps are geometrically forgotten. This is the whole story of Exponentially Weighted Moving Averages.

Level 4 — Synthesis

Exercise 4.1

An RMSprop coordinate is at steady state $E [g^{2}] = 1$ (so effective rate $= η$ ). Suddenly a spike hits: $g = 10$ for one step, then back to $g = 1$ . With $ρ = 0.9$ , $η = 0.1$ , compute $E [g^{2}]$ right after the spike and the effective rate then. Interpret.

Recall Solution

Right after the spike: $E = 0.9 (1) + 0.1 (1 0^{2}) = 0.9 + 10 = 10.9$ . Effective rate $= \frac{0.1}{10.9} \approx \frac{0.1}{3.302} \approx 0.0303$ . The rate dropped to about $30%$ of normal — RMSprop automatically brakes when it hits a steep, spiky direction, then recovers as the average decays back toward $1$ . This built-in damping helps against exploding gradients. See Figure 2.

Exercise 4.2

Adam $=$ RMSprop $+$ momentum $+$ bias correction. Adam's second-moment estimate is $v_{t} = β_{2} v_{t - 1} + (1 - β_{2}) g_{t}^{2}$ — identical in form to RMSprop's $E [g^{2}]$ with $ρ = β_{2}$ . Given constant $g = c$ , starting $v_{0} = 0$ , the bias-corrected estimate is $\overset{v}{^}_{t} = v_{t} / (1 - β_{2}^{t})$ . Show that $\overset{v}{^}_{t} = c^{2}$ for all $t$ (perfect from step 1), whereas raw $v_{t}$ is too small early.

Recall Solution

From 3.3, $v_{t} = (1 - β_{2}) \sum_{k = 1}^{t} β_{2}^{t - k} c^{2} = (1 - β_{2}) c^{2} \sum_{j = 0}^{t - 1} β_{2}^{j} = (1 - β_{2}) c^{2} \cdot \frac{1 - β _{2}^{t}}{1 - β _{2}} = c^{2} (1 - β_{2}^{t}) .$ So $v_{t} = c^{2} (1 - β_{2}^{t}) < c^{2}$ (biased low early — e.g. at $t = 1$ it is only $c^{2} (1 - β_{2})$ ). Bias correction divides by exactly that factor: $\overset{v}{^}_{t} = \frac{v _{t}}{1 - β _{2}^{t}} = \frac{c ^{2} ( 1 - β _{2}^{t} )}{1 - β _{2}^{t}} = c^{2} for all t .$ That is why bias correction exists: RMSprop's raw average underestimates magnitude at the start (see 2.2); Adam corrects it exactly.

Level 5 — Mastery

Exercise 5.1

Design decision. You are training a language model whose embedding rows for rare words receive gradients only occasionally (mostly zero, sometimes large). Would you prefer AdaGrad or RMSprop for those embeddings, and why? Reason with the accumulators.

Recall Solution

For sparse features, AdaGrad shines. Because $G$ only sums when a gradient arrives, a rare word's accumulator stays small, so its effective rate $η / G$ stays large — it takes big steps exactly when it finally gets signal. AdaGrad's "death" needs many nonzero gradients to accumulate, which rare words never provide, so death is not a real problem here. RMSprop's decay would forget a rare word's rare gradients between visits, and its warm-up lag (Ex 2.2) hurts. Verdict: AdaGrad (or a hybrid like Adam) for sparse embeddings; RMSprop for dense layers trained for many steps where AdaGrad would die.

Exercise 5.2

Prove a general bound: for AdaGrad with $ϵ = 0$ , after $t$ steps the effective rate for coordinate $i$ satisfies $\frac{η}{G _{t, i}} \leq \frac{η}{∣ g _{1, i} ∣}$ (assuming $g_{1, i} \neq = 0$ ). Interpret it as "AdaGrad can never step larger than its very first step allows."

Recall Solution

Since $G_{t, i} = \sum_{τ = 1}^{t} g_{τ, i}^{2} \geq g_{1, i}^{2}$ (adding non-negative terms only increases the sum), we have $G_{t, i} \geq ∣ g_{1, i} ∣$ . Taking reciprocals (both sides positive) flips the inequality: $\frac{η}{G _{t, i}} \leq \frac{η}{∣ g _{1, i} ∣} .$ So coordinate $i$ 's effective rate is capped by its step-1 value and only decreases from there. The first gradient permanently sets the ceiling — a reason people sometimes warm up AdaGrad with a small initial accumulator, or reach for schedules instead.

Exercise 5.3

Synthesis with numbers. Compare AdaGrad vs RMSprop total distance travelled by one coordinate over 3 steps with constant $g = 1$ , $η = 0.1$ , $ϵ = 0$ , $ρ = 0.9$ , and (for RMSprop) $E [g^{2}]_{0} = 0$ . Which moves farther, and does the gap widen with more steps?

Recall Solution

AdaGrad ( $G_{t} = t$ ): steps are $\frac{0.1}{1}, \frac{0.1}{2}, \frac{0.1}{3} = 0.1, 0.070711, 0.057735$ . Total $\approx 0.228446$ . RMSprop: $E_{1} = 0.1, E_{2} = 0.19, E_{3} = 0.271$ . Steps $= \frac{0.1}{0.1}, \frac{0.1}{0.19}, \frac{0.1}{0.271} = 0.316228, 0.229416, 0.192095$ . Total $\approx 0.737739$ . RMSprop travels much farther because its accumulator ( $\leq 0.271$ ) is far smaller than AdaGrad's ( $= 3$ ). And by Ex 3.1 vs 3.2, AdaGrad's steps keep shrinking like $1/ t$ while RMSprop's settle near $η / E^{⋆} = 0.1$ — so the gap widens without bound as steps continue. This is the whole reason RMSprop replaced AdaGrad for long training runs.

Recall Self-test summary (cover the right side)

Each line below is written as prompt ::: answer — read the prompt, say your answer aloud, then check it against the text after the :::. AdaGrad accumulator ::: sum of squared grads, $G_{t} = G_{t - 1} + g_{t}^{2}$ , grows forever RMSprop accumulator ::: EWMA, $E_{t} = ρ E_{t - 1} + (1 - ρ) g_{t}^{2}$ , stays bounded AdaGrad effective rate, constant $g = c$ ::: $η / (c t) \to 0$ RMSprop steady effective rate, constant $g = c$ ::: $η / c$ , constant Why bias correction (Adam) ::: raw EWMA is biased low early; divide by $1 - β_{2}^{t}$

Connections

AdaGrad and RMSprop — parent note (theory these exercises drill)
Stochastic Gradient Descent — the one-rate baseline in Ex 2.3
Momentum — the first-moment half of Adam
Adam Optimizer — Ex 4.2 builds it from RMSprop
Exponentially Weighted Moving Averages — the engine behind Ex 3.3
Learning Rate Schedules — alternative referenced in Ex 5.2
Vanishing and Exploding Gradients — the braking in Ex 4.1