Visual walkthrough — Gradient clipping

Before we start, one word we will lean on the whole way:

See Exploding and vanishing gradients for why the arrow sometimes grows monstrously long; this page assumes it can, and fixes it.

---

Step 1 — Draw the gradient as an arrow

WHAT. We take a tiny toy gradient with just two coordinates, $g=(g_x,g_y)$, so it fits on a flat page. Here $g_x$ is the first weight's push, $g_y$ the second's. We plant its tail at the origin and draw an arrow to the point $(g_x,g_y)$.

WHY. Real gradients have millions of coordinates — impossible to draw. But every idea in clipping (direction, length, shrinking) already appears in 2‑D. If you understand two numbers, you understand a million: the math is identical, just longer.

PICTURE. The blue arrow below is the gradient. Its horizontal reach is $g_x$, its vertical reach is $g_y$.

---

Step 2 — Measure the arrow's length (the norm)

WHAT. We need a single number for "how long is this arrow?" The arrow, its horizontal shadow $g_x$, and its vertical shadow $g_y$ form a right triangle — a triangle with one square corner. The two shadows are the short sides; the arrow is the slanted long side (the hypotenuse).

WHY. Clipping's whole job is "is the arrow too long?" We literally cannot ask that until we can measure length. And "too long" must mean the straight-line length of the arrow, not $g_x$ or $g_y$ alone — because the arrow can be short in each coordinate yet long overall (think $g_x=g_y=100$).

Which tool, and why this one? To get the slanted side of a right triangle from its two square sides, the tool is the Pythagorean theorem: for a right triangle, $(\text{long side}{)}^2=({\text{side}}_1{)}^2+({\text{side}}_2{)}^2$. We use it and not, say, simple addition, because $g_x+g_y$ would over-count — walking east then north is shorter as a diagonal than as two separate legs. Squaring, adding, then square-rooting is exactly the straight-line distance.

PICTURE. The triangle: red horizontal leg $g_x$, green vertical leg $g_y$, blue hypotenuse = the arrow, with its length $\Vert g\Vert$ labelled on the slant.

---

Step 3 — Draw the safe zone (the threshold circle)

WHAT. We pick one number $c$, the threshold, and declare: any arrow of length $\le c$ is safe. All safe arrow-tips live inside (or on) a circle of radius $c$ centred at the origin. That circle is the "leash length."

WHY. We need a fixed line between "fine" and "dangerous," decided before we see the arrow, so the rule is objective. The circle is the natural boundary because "length $\le c$" is exactly the definition of a disk of radius $c$ — every safe arrow fits inside, every unsafe one pokes out.

PICTURE. A yellow circle of radius $c$. A short blue arrow sits inside (safe); a long blue arrow pierces the circle (unsafe). The tip is all that matters — inside = keep, outside = must fix.

---

Step 4 — The safe case: do nothing

WHAT. Suppose $\Vert g\Vert \le c$. The arrow already lives inside the circle. We leave it completely untouched: $\hat{g}=g$. (The hat $\hat{g}$ just means "the arrow after clipping.")

WHY. The arrow's direction is our best guess at downhill and its length is reasonable — distorting a perfectly good gradient would only slow learning for no reason. Rule one of any safety device: don't fire when there's no danger. See Loss landscapes and cliffs for why most ordinary steps are exactly this harmless case.

PICTURE. Arrow inside the circle, untouched; a green check marks "keep as-is."

---

Step 5 — The dangerous case: same direction, shrink the length

WHAT. Now $\Vert g\Vert >c$: the arrow pokes out of the circle. We build a replacement $\hat{g}$ obeying two demands at once:

1. same direction as $g$ — so $\hat{g}=\alpha g$ for some positive number $\alpha$ (multiplying an arrow by a positive number keeps its heading, only rescales its length);
2. length exactly $c$ — sitting right on the leash.

WHY. Near a cliff, the arrow's length is untrustworthy (it exploded) but its direction is still roughly the true downhill. So we surgically keep the direction and cut the length back to the safe boundary $c$ — no shorter (wasteful), no longer (unsafe).

Solve for the shrink factor. Demand 2 says the new length is $\Vert \hat{g}\Vert =\Vert \alpha g\Vert =\alpha \Vert g\Vert$, and we want that to equal $c$: $\underset{\text{new length}}{\underbrace{\alpha \Vert g\Vert }}=\underset{\text{leash}}{\underbrace{c}}⟹\alpha =\frac{c}{\Vert g\Vert }.$ Since $\Vert g\Vert >c$, this $\alpha$ is a number between 0 and 1 — a genuine shrink.

PICTURE. The long arrow, and along the same ray, a shorter arrow whose tip lands exactly on the yellow circle. The scale factor $\alpha =c/\Vert g\Vert$ is annotated on the ray.

---

Step 6 — Fuse both cases into one formula

WHAT. We now merge "do nothing" (Case 1) and "scale by $c/\Vert g\Vert$" (Case 2) into a single expression, using the $\min$ (minimum: "pick the smaller of two numbers").

WHY. Two if-branches are error-prone; one clean formula runs the same code path every step. The trick: notice the branches only differ in the multiplier — Case 1 multiplies by $1$, Case 2 by $c/\Vert g\Vert$. We want whichever is smaller.

• If $\Vert g\Vert \le c$: then $c/\Vert g\Vert \ge 1$, so $\min (1,c/\Vert g\Vert )=1$ → do nothing. ✓ (matches Step 4)
• If $\Vert g\Vert >c$: then $c/\Vert g\Vert <1$, so $\min (1,c/\Vert g\Vert )=c/\Vert g\Vert$ → shrink. ✓ (matches Step 5)

The $\min$ automatically selects the right branch. That is the whole reason it appears.

PICTURE. A number line of the multiplier value $c/\Vert g\Vert$: to the left of $1$ (unsafe, we shrink), to the right of $1$ (safe, $\min$ locks us to $1$). The kink at $\Vert g\Vert =c$ is where behaviour switches.

---

Step 7 — Edge & degenerate cases (never leave the reader stranded)

WHAT & WHY. A formula is only trustworthy if it survives the weird inputs. We test the corners.

• Exactly on the boundary, $\Vert g\Vert =c$. Then $c/\Vert g\Vert =1$, $\min (1,1)=1$ → unchanged. Good: the boundary is "safe," no jolt.
• The zero gradient, $g=(0,0)$. Length $\Vert g\Vert =0$, so $c/\Vert g\Vert$ would divide by zero — undefined! But a zero arrow has no direction and no danger, so the correct answer is simply $\hat{g}=0$. In real code this is handled by adding a tiny $ϵ$ or checking $\Vert g\Vert >0$ first. Never divide by a zero norm.
• Already tiny, $\Vert g\Vert \ll c$. Factor $c/\Vert g\Vert$ is huge $(>1)$, $\min$ clamps it to $1$ → untouched. Clipping never grows a small gradient — which is exactly why it cannot cure vanishing gradients (that needs Weight initialization, Batch Normalization, or gating from Recurrent Neural Networks).
• One coordinate zero, e.g. $g=(0,5)$. Fine — the norm is just $5$, and scaling keeps that lone axis aligned. No special handling.

PICTURE. Four mini-panels: on-boundary (green tick), zero arrow (a dot, "undefined → set 0"), tiny arrow (untouched), axis-aligned arrow (untouched-or-scaled cleanly).

Recall Quick self-check:

$g=(0,5,12)$, $c=6.5$ Length $=\sqrt{0+25+144}=\sqrt{169}=13$. Unsafe since $13>6.5$. Factor $=6.5/13=0.5$. So $\hat{g}=0.5\cdot (0,5,12)=(0,2.5,6)$, whose length is $\sqrt{0+6.25+36}=\sqrt{42.25}=6.5$. Exactly on the leash. ✓

---

The one-picture summary

WHAT. One diagram compresses the entire derivation: safe arrows stay put; unsafe arrows slide down their own ray until their tip rests on the circle of radius $c$.

Every clipped arrow lands on the yellow circle along its original direction — "same arrow, shorter arrow." Every safe arrow is left exactly where it was. That single behaviour is the boxed formula, and the boxed formula is that single behaviour. For where this sits in the training loop (after Backpropagation through time, before the Optimizers - SGD Adam RMSProp step), return to the parent Gradient clipping.

Recall Feynman retelling — the whole walkthrough in plain words

A gradient is just an arrow: which way to nudge the weights, and how hard (Step 1). To ask "is it too big?" we first need its length, which is the slanted side of the right triangle its coordinates make — Pythagoras, square-add-root (Step 2). We draw a circle of radius $c$ around the origin: arrows inside are safe, arrows poking out are dangerous (Step 3). If the arrow is inside, we touch nothing — a good gradient shouldn't be spoiled (Step 4). If it pokes out, we keep its heading and slide its tip back onto the circle by multiplying the whole arrow by $c/\Vert g\Vert$, a number less than one (Step 5). Both rules fuse into one line using $\min$, which quietly picks "$1$" when safe and "$c/\Vert g\Vert$" when not (Step 6). We double-check the corners: on the boundary nothing happens, a zero arrow we just leave at zero (never divide by zero), and a tiny arrow is never inflated — so clipping fights explosions, never rescues vanishing (Step 7). One sentence to keep: same arrow, shorter arrow.

---

Connections

• Gradient clipping (parent)
• Exploding and vanishing gradients
• Loss landscapes and cliffs
• Recurrent Neural Networks
• Backpropagation through time
• Optimizers - SGD Adam RMSProp
• Weight initialization
• Batch Normalization