Foundations — Gradient clipping

This page assumes nothing. Before you can read the parent note Gradient Clipping, you need to be fluent with about a dozen little symbols. We build each one from a picture, in an order where every new idea only uses the ones before it.

0. The picture the whole topic lives in

Everything in this topic happens on a loss landscape: a surface where the height tells you how wrong the network currently is. Low = good, high = bad. Training = walking downhill.

1. Numbers, then a list of numbers: the vector

Picture: a vector is an arrow. Start at the origin $(0,0)$; the numbers tell you how far to go right and up. So $(3,4)$ means "3 right, 4 up", and the arrow points to that spot.

Why the topic needs it: the gradient is a vector — one number per adjustable knob (parameter) in the network. A tiny network might have 2 knobs; a real one has millions. But the ideas work identically, so we practice with 2- and 3-number vectors you can draw.

We write a vector's individual numbers using a subscript (a small index): if $g=(g_1,g_2)$, then $g_1$ is the first number and $g_2$ the second. The subscript is just a name tag, nothing more.

2. How long is an arrow? The norm $\Vert g\Vert$

The single most important quantity in this whole topic is: how long is the gradient arrow? A short arrow = small safe step. A giant arrow = dangerous leap.

Why this exact formula and not something else? We want ordinary ruler distance, and ruler distance in a right triangle is the Pythagorean theorem: the diagonal is the square root of the sum of the squares of the sides. That is the reason the square-root-of-squares shows up.

3. Scaling an arrow: multiply a vector by a scalar $\alpha$

Picture: the arrow keeps pointing the same way but its length changes. $\alpha =0.5$ makes it half as long; $\alpha =2$ makes it twice as long. If $\alpha >0$ the direction is untouched — this is the whole secret of clip-by-norm: it multiplies the gradient by one positive scalar, so the arrow's direction survives and only its length shrinks.

The key fact we will use: because scaling multiplies length by $\alpha$, $\Vert \alpha g\Vert =\alpha \Vert g\Vert (\text{for }\alpha \ge 0).$ Double the arrow, double its length. This is why choosing $\alpha =c/\Vert g\Vert$ later makes the new length exactly $c$.

4. Two comparison tools: $\min$ and $\mathrm{clip}$

The parent formula is $\hat{g}=g\cdot \min (1,c/\Vert g\Vert )$. Two mini-functions live in there.

Why it appears: we want to shrink an oversized arrow but never stretch a safe one. Taking the minimum with $1$ guarantees the scale factor is at most $1$ (never enlarges) — one formula handles "already safe" and "too big" at once.

Picture: imagine two fences at heights $+v$ and $-v$. Any value poking past a fence gets pushed back to it. This is clip-by-value — it treats each number separately, which (as the parent warns) can bend the arrow's direction.

The hat notation $\hat{g}$ (little "^" over the letter) just means "the modified $g$ after clipping". It's a fresh name for the output, not a new operation.

5. Where the gradient comes from: derivative and $\frac{\partial L}{\partial w}$

Picture: stand on the loss surface and tilt your head along one knob's direction. The steepness you feel is the derivative. Steep downhill = big negative slope; flat = zero slope.

Why the topic needs it: the gradient is literally the collection of these slopes, one per knob. That is what tells us which way is downhill and how steep.

So the gradient we clip is exactly this steepness-vector. When it's short, all slopes are mild. When it's enormous (near a cliff), the length is untrustworthy but the direction still roughly points uphill — which is why we keep direction and trim length.

6. The update rule and its symbols: $w\leftarrow w-\eta g$

Picture: you're the dot on the surface. $-g$ is the downhill arrow; $\eta$ scales it to a comfortable stride; you move there. Repeat.

7. Why gradients explode: the product of many factors

Why the topic needs it: a deep network passes signals through many layers, and backpropagation multiplies one factor per layer. If each factor is a bit bigger than $1$, the product grows like $1.5^n$ — explosively with depth. That geometric blow-up is why gradients explode. (The parent note calls each factor a "Jacobian singular value"; for D1 you only need: many numbers bigger than 1, multiplied together, get huge fast.)

8. Putting the symbols together (preview of the parent)

You can now read the boxed formula fully: $\hat{g}=g\cdot \min \left(1,\frac{c}{\Vert g\Vert }\right)$

• $g$ — the gradient vector (§1).
• $\Vert g\Vert$ — its length via Pythagoras (§2).
• $c$ — a chosen threshold (the maximum allowed length).
• $c/\Vert g\Vert$ — the scalar that would shrink the length to exactly $c$ (§3, since $\Vert \alpha g\Vert =\alpha \Vert g\Vert$).
• $\min (1,\cdot )$ — "shrink if too big, else leave alone" (§4).
• $\hat{g}$ — the safe, trimmed gradient handed to the update rule (§6).

Every piece is now earned. Head to the parent note for the full derivation and worked cases.

Prerequisite map

Equipment checklist

Cover the right side and test yourself — you're ready for the parent note when all pass.

Connections

• Parent: Gradient Clipping
• Exploding and vanishing gradients
• Backpropagation through time
• Recurrent Neural Networks
• Optimizers - SGD Adam RMSProp
• Weight initialization
• Batch Normalization
• Loss landscapes and cliffs