Worked examples — AdaGrad and RMSprop

This is the worked-example lab for the parent note. There we derived the two update rules. Here we run them through every case class — tiny gradients, huge gradients, gradients that flip sign, zero gradients, the long-run limit, a real word problem, and an exam-twist. Guess each answer before you read the steps.

A quick reminder of the two rules we are stress-testing (both act on one coordinate $i$, so every $g$, $G$, $E$ below is a plain number, not a vector):

---

The scenario matrix

Every situation these optimizers face falls into one of these cells. The examples below are tagged with the cell(s) they cover.

Cell	Scenario	What we must check
C1	Large gradient vs small gradient (two coordinates)	the equalizing effect — does a small-$g$ coord get a big step?
C2	Sign of $g$ flips ($+\to -$)	update direction flips, but the divisor uses $g^2$ so it ignores sign
C3	Constant gradient, long run	AdaGrad step $\to 0$; RMSprop step $\to \eta$
C4	$g=0$ (dead / flat direction)	update = 0, but does the divisor keep its memory?
C5	First step, empty accumulator	$ϵ$ saves us from dividing by zero
C6	Sudden gradient spike (exploding-ish)	how each optimizer damps a burst — links to Vanishing and Exploding Gradients
C7	Real-world word problem	translate an English scenario into the update
C8	Exam twist: choose $\rho$ for a memory horizon	connect $\rho$ to "how many steps back it remembers"

---

Example 1 — the equalizing effect · Cell C1

Forecast: guess — will B (with the tiny gradient) take a tiny step, or does AdaGrad rescue it?

1. A: $G_A=1^2=1$, so step $=\frac{0.1}{\sqrt{1}}\cdot 1=0.1$. Why this step? First gradient sets A's own scale — dividing $1$ by $\sqrt{1}=1$ leaves the raw learning rate untouched.
2. B: $G_B=0.1^2=0.01$, so step $=\frac{0.1}{\sqrt{0.01}}\cdot 0.1=\frac{0.1}{0.1}\cdot 0.1=0.1$. Why this step? B divides by its own small $\sqrt{G}=0.1$, which cancels B's small gradient. The tiny gradient is boosted back up.

Result: both move $0.1$. B's raw gradient was $10\times$ smaller yet it takes an equal step. This is the whole point — see the parent's "fog / cliffs and slopes" picture.

Verify: $\frac{0.1}{\sqrt{0.01}}\cdot 0.1=\frac{0.1}{0.1}\cdot 0.1=0.1$. Units: $\eta$ and gradient are in "parameter per step" balance so the step is a plain parameter displacement. ✓

---

Example 2 — the sign of the gradient · Cell C2

Forecast: does flipping the sign change how big the step is, or only which way it goes?

1. Step 1: $E_1=0.9(0)+0.1(2^2)=0.4$. Step $=\frac{0.1}{\sqrt{0.4}}\cdot 2=\frac{0.2}{0.6325}\approx 0.3162$. Why this step? The divisor $\sqrt{E}$ only sees $g^2=4$; the positive step moves $\theta$ down (subtracted).
2. Step 2: $E_2=0.9(0.4)+0.1((-2{)}^2)=0.36+0.4=0.76$. Step $=\frac{0.1}{\sqrt{0.76}}\cdot (-2)=\frac{-0.2}{0.8718}\approx -0.2294$. Why this step? $g^2=(-2{)}^2=4$ again, so the divisor grew ($E$ rose to $0.76$), and the sign of the update flipped because $g$ is now negative.

Lesson: the divisor is sign-blind (it squares $g$), but the numerator carries the sign — so direction reverses while magnitude is governed only by accumulated size.

Verify: $\frac{0.1}{\sqrt{0.4}}\cdot 2\approx 0.31623$ and $\frac{0.1}{\sqrt{0.76}}\cdot (-2)\approx -0.22942$. ✓

---

Example 3 — the long run: AdaGrad dies, RMSprop lives · Cell C3

Forecast: which one is still moving at step 100?

1. AdaGrad, step $t$: since $g=1$ always, $G_t={\sum }_{\tau =1}^{t}1=t$. Step $=\frac{0.1}{\sqrt{t}}$.
   • $t=1$: $0.1$. $t=100$: $\frac{0.1}{10}=0.01$. $t\to \infty$: $\to 0$. Why? The sum only grows, so the denominator grows without bound — the parameter freezes.
2. RMSprop steady state: solve $E=0.9E+0.1(1)\Rightarrow 0.1E=0.1\Rightarrow E=1$. Step $=\frac{0.1}{\sqrt{1}}=0.1$ forever. Why? The EWMA saturates at the true mean-square $g^2=1$; a bounded divisor gives a constant step.

Result: at step 100 AdaGrad crawls at $0.01$ while RMSprop still strides at $0.1$ — a $10\times$ gap, growing forever. This is the parent note's "Accumulates → Arrests" vs "Remembers Recently → Runs."

Verify: AdaGrad$(100)=0.1/\sqrt{100}=0.01$; RMSprop steady state $E=1$, step $=0.1$. ✓

---

Example 4 — a dead direction: $g=0$ · Cell C4

Forecast: does the parameter move? Does the optimizer forget its accumulated memory?

1. Update: step $=\frac{\eta }{\sqrt{E_t}+ϵ}\cdot 0=0$. Why? Zero gradient in the numerator kills the whole step — no gradient, no reason to move. (Momentum, by contrast, would keep coasting; see Momentum.)
2. New accumulator: $E_t=0.9(0.5)+0.1(0^2)=0.45$. Why? A zero-gradient step is still an observation of "recent size $=0$," so the average decays from $0.5$ toward $0$ — the memory of past bumpiness fades but is not erased.

Lesson: $g=0$ ⇒ no move, but the divisor shrinks toward the recent mean-square. If gradients stay zero forever, $E\to 0$ and the next nonzero gradient would get a very large effective step (that's what $ϵ$ ultimately bounds).

Verify: update $=0$ exactly; $E_t=0.9\cdot 0.5+0.1\cdot 0=0.45$. ✓

---

Example 5 — the very first step and the role of $ϵ$ · Cell C5

Forecast: with $G_0=0$ and $g_1=0$, does the formula explode ($\frac{0}{0}$)?

1. Step 1: $G_1=0+0^2=0$. Step $=\frac{0.01}{\sqrt{0}+10^{-8}}\cdot 0=\frac{0.01}{10^{-8}}\cdot 0=0$. Why $ϵ$ matters: without $ϵ$ we'd write $\frac{0.01}{0}$, undefined. With $ϵ$ the denominator is $10^{-8}$ — finite — and the numerator's $g=0$ makes the whole step $0$ anyway. $ϵ$ turns a "$0/0$" into a clean "$0$."
2. Step 2: $G_2=0+3^2=9$. Step $=\frac{0.01}{\sqrt{9}+10^{-8}}\cdot 3=\frac{0.01}{3.00000001}\cdot 3\approx 0.01$. Why? $\sqrt{9}=3$ cancels the gradient $3$, leaving $\approx \eta$. The $ϵ$ is negligible here — it only ever matters when the accumulator is tiny.

Verify: step 1 $=0$; step 2 $=\frac{0.01}{3+10^{-8}}\cdot 3\approx 0.0099999999\approx 0.01$. ✓

---

Example 6 — a gradient spike (near-exploding) · Cell C6

Forecast: SGD would take a step of $\eta \cdot g=0.1\cdot 10=1$ (huge!). Does RMSprop take a $10\times$ bigger step too?

1. Update $E$ with the spike: $E=0.9(1)+0.1(10^2)=0.9+10=10.9$. Why? The squared spike $100$ enters the average, but weighted only by $(1-\rho )=0.1$, so $E$ jumps to $10.9$, not all the way to $100$.
2. RMSprop step: $\frac{0.1}{\sqrt{10.9}}\cdot 10=\frac{1}{3.3015}\approx 0.3029$. Why? Because the divisor $\sqrt{E}$ grew with the spike, the $10\times$-bigger gradient produces only a $\approx 3\times$-bigger step, not $10\times$. RMSprop self-damps bursts.
3. Plain SGD step: $\eta \cdot g=0.1\cdot 10=1.0$. Why the contrast? SGD has no divisor, so the spike passes straight through — a jump of $1.0$ that can destabilize training. This is exactly the exploding-gradient danger in Vanishing and Exploding Gradients that per-coordinate scaling tames.

Verify: $E=10.9$; RMSprop step $=\frac{0.1}{\sqrt{10.9}}\cdot 10\approx 0.30289$; SGD step $=1.0$. ✓

---

Example 7 — real-world word problem · Cell C7

Forecast: which rare word gets the bigger update when it finally appears?

1. "the": hit 10,000 times with $g=1$, so $G_{\text{the}}=10000$. Step $=\frac{0.5}{\sqrt{10000}}\cdot 1=\frac{0.5}{100}=0.005$. Why? A frequent word has accumulated a huge $G$; its effective rate is now tiny — it has already learned, so it fine-tunes.
2. "aardvark": hit once with $g=1$, so $G_{\text{aardvark}}=1$. Step $=\frac{0.5}{\sqrt{1}}\cdot 1=0.5$. Why? A rare word has a tiny accumulator, so its one appearance earns a full-sized step — it learns fast from scarce data.

Result: the rare word moves $100\times$ farther on its update. This is why AdaGrad shines on sparse NLP problems (the parent's "good for convex / sparse" note).

Verify: "the" step $=0.5/\sqrt{10000}=0.005$; "aardvark" step $=0.5/\sqrt{1}=0.5$; ratio $=100$. ✓

---

Example 8 — exam twist: pick $\rho$ for a memory horizon · Cell C8

Forecast: guess $\rho$ before computing.

1. Solve the horizon: $\frac{1}{1-\rho }=100\Rightarrow 1-\rho =0.01\Rightarrow \rho =0.99$. Why this formula? In an EWMA the weight on the gradient $k$ steps ago is $(1-\rho ){\rho }^k$; these weights sum to $1$ and their "center of mass" (effective window) is $\frac{1}{1-\rho }$ steps. Longer memory ⇒ $\rho$ closer to $1$.
2. Weight on the newest gradient: the newest term carries weight $(1-\rho )=0.01=1\%$. Why so small? A long memory means each single step contributes little — the estimate is smooth, changing slowly. (Contrast the default $\rho =0.9$: window $\approx 10$ steps, newest weight $10\%$.)

Lesson: $\rho$ is a memory dial, not a step-size dial — that's the parent's "$\rho$ vs $\eta$" mistake warning. Bigger $\rho$ = smoother, longer-memory divisor.

Verify: $1/(1-0.99)=100$; newest weight $=1-0.99=0.01$. ✓

---

Recall Which cell did each example cover?

Ex1 equalizing (C1) ::: small vs large gradient get equal steps Ex2 sign flip (C2) ::: divisor squares $g$, only numerator carries sign Ex3 long run (C3) ::: AdaGrad $\to 0$, RMSprop $\to \eta$ Ex4 dead direction (C4) ::: $g=0$ ⇒ no move, but $E$ decays Ex5 first step + $ϵ$ (C5) ::: $ϵ$ turns $0/0$ into a clean $0$ Ex6 spike (C6) ::: RMSprop self-damps a burst; SGD lets it through Ex7 word problem (C7) ::: rare word gets a huge step, frequent word a tiny one Ex8 exam twist (C8) ::: $\rho =0.99$ for a 100-step memory window

Recall Quick self-test

Constant $g=1$, AdaGrad, $\eta =0.1$: step at $t=100$? ::: $0.1/\sqrt{100}=0.01$ Same but RMSprop steady state? ::: $0.1$ (E saturates at 1) RMSprop $\rho$ for a 20-step memory window? ::: $1-\rho =1/20=0.05\Rightarrow \rho =0.95$ If $g_t=0$ one step, does RMSprop's $E$ stay the same? ::: No — it decays: $E_t=\rho E_{t-1}$

Connections

• AdaGrad and RMSprop — the parent note these examples drill
• Stochastic Gradient Descent — the un-scaled baseline (Ex6 comparison)
• Momentum — would keep moving on a $g=0$ step (Ex4 contrast)
• Adam Optimizer — RMSprop's divisor + momentum's numerator, combined
• Exponentially Weighted Moving Averages — the EWMA math behind Ex3 & Ex8
• Learning Rate Schedules — hand-tuned decay vs the automatic decay in AdaGrad
• Vanishing and Exploding Gradients — the spike-damping in Ex6