AdaGrad and RMSprop: Foundations — AI-ML deep dive

This page builds every symbol used in the parent note from absolute zero. If you have never seen a gradient, a subscript, or a square-root-in-a-denominator, start at line one and don't skip.

1. A parameter $θ$ — the knob we are turning

Picture it: imagine a mixing board with many sliders. Each slider is one $θ$ . Push it up or down and the network's predictions change.

Why the topic needs it: the whole point of AdaGrad/RMSprop is to decide how far to push each slider on each step. No parameter, nothing to update.

2. The subscripts $i$ and $t$ — which knob, when

There are two labels stuck onto every symbol. They answer two different questions.

So $θ_{t, i}$ reads: "the value of slider number $i$ at time-step $t$ ." Two labels, two totally separate meanings — don't blend them.

Figure 1 (below) lays the sliders out as a grid: move right and time $t$ advances; move up and you pick a different coordinate $i$ . The yellow dot is one specific $θ_{t, i}$ — one slider frozen at one moment. Use it to see that the two subscripts point along two different axes.

Why the topic needs it: AdaGrad's whole trick is a per-coordinate step, so we must be able to talk about coordinate $i$ alone. And because the step depends on history, we must track time $t$ .

3. Loss $L$ — the "how wrong are we" number

Picture it: $L$ is the height of a landscape. The sliders $θ$ are your east–west, north–south position. Walk to the lowest valley → smallest loss.

4. The gradient $g$ and the symbol $\nabla$ — which way is downhill

Here is the first genuinely new tool. Why do we need it?

Picture it: stand on a hillside. The gradient is an arrow pointing in the steepest uphill direction; its length is how steep. To go down, we step in the opposite direction — that minus sign is where the " $-$ " in every update rule comes from.

Figure 2 (below) draws the loss as a bowl seen from above (the thin white rings are equal-height contours). The yellow dot is your position $θ_{t}$ ; the pink arrow is $g_{t}$ pointing steepest uphill; the blue arrow is the actual step $- η g_{t}$ pointing downhill toward the star (the valley). It shows exactly why we subtract the gradient.

Why the topic needs it: $g_{t, i}$ is the raw material. AdaGrad/RMSprop never change the direction $g$ points — they only rescale how big a step to take. See Stochastic Gradient Descent for where these gradients come from (from mini-batches, which makes them noisy — a reason we later want to average their sizes).

5. The learning rate $η$ — how big a step

Picture it: $η$ is your stride length. Too big → you leap over the valley and bounce around. Too small → you inch along and never arrive. See Learning Rate Schedules for changing $η$ over time as an alternative to what AdaGrad does automatically.

Why the topic needs it: AdaGrad and RMSprop take this single global $η$ and secretly turn it into a different effective stride for every slider. That is their entire job.

6. Squaring and the square root — turning slopes into "sizes"

The rules square the gradient and later take a square root. Why both?

Figure 3 (below) compares three curves as the gradient $g$ runs left-to-right: blue $g$ keeps its sign, pink $∣ g ∣$ gives size only, yellow $g^{2}$ gives size and punishes big slopes (it shoots up fast). The two yellow dots mark $(- 3)^{2} = 9 = 3^{2}$ — proof that squaring throws the sign away.

Why the topic needs it: the accumulators $G$ and $E [g^{2}]$ (next) are built from $g^{2}$ ; the update divides by of them. Averaging $g^{2}$ is a mean-square measure of gradient magnitude. (Careful: the true statistical variance is $E [g^{2}] - (E [g])^{2}$ ; our $E [g^{2}]$ is only the raw mean of the squares, which equals the variance only when the average gradient $E [g]$ is zero. The parent note scales by this raw second moment, not by the variance.) This contrasts with Momentum, which averages the plain gradient (the first moment).

7. The accumulator $G_{t, i}$ — remembering all past slope-sizes

Picture it: a bucket per slider. Every step you pour in the current squared slope. The bucket only fills — it never empties.

Why the topic needs it (and its fatal flaw): dividing $η$ by $G$ shrinks the step for sliders that have seen big/frequent slopes. But because the bucket only grows, $G_{t, i} \to \infty$ and the step $\to 0$ : AdaGrad eventually freezes. That flaw is the reason RMSprop exists.

8. The EWMA $E [g^{2}]_{t, i}$ — remembering only recent slope-sizes

Picture it: the same bucket, but with a small hole in the bottom. Each step you pour in a little new water and some old water leaks out. The level stops rising and holds steady at the recent average. The full mechanics live in Exponentially Weighted Moving Averages.

Figure 4 (below) runs both accumulators on a constant slope $g = 1$ . The pink line is AdaGrad's sum $G_{t}$ : it climbs forever ( $\to \infty$ ), so its step dies. The blue line is RMSprop's EWMA $E [g^{2}]_{t}$ : it settles onto the yellow dashed line at $1$ and stays there — a bounded memory that keeps the step alive.

Why the topic needs it: replacing $G$ (a sum) with $E [g^{2}]$ (a bounded average) is the single change from AdaGrad to RMSprop. Because a bounded average never blows up, the step never dies.

9. The tiny number $ϵ$ — a safety cushion

Picture it: a rubber bumper stopping the denominator from ever touching $0$ .

10. Putting the symbols together

Everything the parent note writes is now just these pieces assembled:

Combine RMSprop's per-slider stride with Momentum's averaged direction and you get the Adam Optimizer. Per-slider scaling also tames wildly different gradient magnitudes — the subject of Vanishing and Exploding Gradients.

Prerequisite map

How to read this diagram: each box is one idea from this page; an arrow $X \to Y$ means "you need $X$ before $Y$ makes sense" (X feeds into Y). Start at the top boxes with no incoming arrows and follow the arrows downward — that is a valid order to learn these in. The two bottom-most boxes are AdaGrad and RMSprop, which everything else builds toward.

Equipment checklist

I can explain what a parameter

θ

is in plain words

An adjustable number (a slider) inside the network; training picks its value.

I know what the two subscripts in

θ_{t, i}

mean

t

= time-step (which update),

i

= coordinate (which parameter). Same slider, next moment when

t

advances.

I can say what the loss

L

is and picture it

A single "how wrong" number; picture it as the height of a landscape we want to minimise.

I know what the gradient

g_{t} = \nabla_{θ} L (θ_{t})

The list of slopes of

L

measured at the current parameters

θ_{t}

— an arrow pointing steepest uphill; we step opposite to go down, hence the minus sign.

I can read the sign of

g_{t, i}

Positive → decrease

θ_{i}

; negative → increase

θ_{i}

; zero → flat, no move.

I know what the learning rate

η

controls

The overall step size (stride length); too big overshoots, too small crawls.

I know why we square the gradient

To drop the sign and punish large slopes more;

later restores the original scale.

I can write the AdaGrad accumulator with its base case

G_{0, i} = 0

and

G_{t, i} = \sum_{τ = 1}^{t} g_{τ, i}^{2} = G_{t - 1, i} + g_{t, i}^{2}

— a bucket that only fills.

I know why AdaGrad eventually freezes

G

only grows, so

G \to \infty

and the effective step

\to 0

I can write the RMSprop EWMA with its base case

E [g^{2}]_{0, i} = 0

and

E [g^{2}]_{t, i} = ρE [g^{2}]_{t - 1, i} + (1 - ρ) g_{t, i}^{2}

— a bounded, leaky average.

I know the difference between

ρ

and

η

ρ

= memory length of the squared-gradient average;

η

= overall step size.

I know why

ϵ

is there

To prevent division by zero when the accumulator is tiny.

I know why

E [g^{2}]

is the raw second moment, not the variance

Variance is

E [g^{2}] - (E [g])^{2}

; our

E [g^{2}]

is just the mean of the squares, equal to variance only when the mean gradient is zero.