Gradient clipping: Worked examples — AI-ML deep dive

Before anything, let us re-anchor the two tools in plain words so every symbol below is earned.

Definition The two clip rules, in words

$g$ = the gradient vector: a list of numbers, one per parameter. Think of it as an arrow pointing in the direction the loss climbs fastest.
$∥ g ∥$ = the L2 norm = the length of that arrow $= g_{1}^{2} + g_{2}^{2} + \dots$ . It answers "how far would one raw step throw us?"
$c$ = the norm threshold: the longest arrow we allow.
$v$ = the value threshold: the biggest single component we allow.
$α$ = the scale factor: the single positive number we multiply the whole gradient by. For clip-by-norm we define $α = min (1, \frac{c}{∥ g ∥})$ .
Clip-by-norm: $\overset{g}{^} = α g = g \cdot min (1, \frac{c}{∥ g ∥})$ — same arrow, maybe shorter.
Clip-by-value: $\overset{g}{^}_{i} = clip (g_{i}, - v, v) = max (- v, min (g_{i}, v))$ — cap each number on its own.

Clipping produces the safe gradient $\overset{g}{^}$ ; whatever the optimizer does with it afterwards is a separate step we don't need here.

The scenario matrix

Every clipping problem is one of these cells. The examples below each carry a tag like (A2).

#	Cell class	What makes it special	Covered by
A1	Norm, safe gradient	$∥ g ∥ \leq c$ → no change	Ex 1
A2	Norm, exploding gradient	$∥ g ∥ > c$ → shrink	Ex 2
A3	Norm, exactly on the leash	$∥ g ∥ = c$ (boundary)	Ex 3
A4	Norm, negative components	signs must survive scaling	Ex 4
B1	Value clip, mixed signs	each coord clamped to $[- v, v]$	Ex 5
B2	Value vs norm on same $g$	direction distortion shown	Ex 6
C1	Zero / degenerate gradient	$∥ g ∥ = 0$ , division danger	Ex 7
C2	Scalar (1-D) gradient	norm = absolute value	Ex 7
D1	Real-world word problem	RNN loss spike, pick behaviour	Ex 8
D2	Exam twist	solve for the threshold $c$	Ex 9

Look at the figure: the black arrow is the raw $g$ of length $5$ . The red arrow is $\overset{g}{^}$ — it lands exactly on the dashed circle of radius $c = 2$ , lying along the same ray. The circle is the "leash": any raw arrow poking outside it gets pulled back onto its rim.

Ex 3 — exactly on the leash (A3)

Ex 4 — negative components survive scaling (A4)

Ex 5 — clip-by-value, mixed signs (B1)

Ex 6 — same vector, two rules, direction contrast (B2)

Worked example Statement

$g = (6, 8)$ , with $c = v = 5$ . Do it both ways and compare.

Forecast: One method keeps the arrow pointing at "3:4"; the other tilts it. Predict which — and predict whether the value-clipped result even fits inside the norm budget.

By norm: $∥ g ∥ = 36 + 64 = 10$ , $α = 5/10 = 0.5$ , $\overset{g}{^}_{norm} = (3, 4)$ . Why? Scalar scaling → direction $6 : 8 = 3 : 4$ preserved.
Length check: $3^{2} + 4^{2} = 9 + 16 = 5 = c$ . Why this step? We must confirm the norm-clipped vector actually lands on the leash (length exactly $c$ ) — this is the guarantee that separates it from value clipping, so we check it before comparing.
By value: $\overset{g}{^}_{val} = (min (6, 5), min (8, 5)) = (5, 5)$ . Why? Both coords exceed $5$ , both pinned to $5$ .
Its length: $25 + 25 = 50 \approx 7.07$ . Why compute it? To expose a trap.

Verify: The value-clipped vector has length $\approx 7.07 > 5$ — it still overshoots the norm budget and its direction is now $1 : 1$ instead of $3 : 4$ . Clip-by-norm respects the geometry; clip-by-value does not. ✓

In the figure the black arrow is $g = (6, 8)$ . The red arrow is the norm-clipped result sitting on the ray at length $5$ ; the grey arrow is the value-clipped $(5, 5)$ — visibly rotated away from the original ray. Same start, different destinations.

Ex 7 — zero gradient & scalar gradient (degenerate) (C1, C2)

Worked example Statement

Two edge inputs, each solved on its own numbered track. (a) $g = (0, 0, 0)$ with $c = 2$ . (b) a single scalar gradient $g = - 7$ with $c = 4$ .

Forecast: In (a) the formula has $c /∥ g ∥ = 2/0$ . Does clipping blow up? In (b), what even is the norm of one number?

Track (a) — the zero gradient:

$∥ g ∥ = 0^{2} + 0^{2} + 0^{2} = 0 = 0$ . Naively $c /∥ g ∥ = 2/0$ is undefined. Why does this matter? A raw division would crash.
Test the shrink condition: is $∥ g ∥ > c$ ? Here $0 > 2$ is false, so we take the safe branch and never divide. Why this step? The rule only reaches the division when $∥ g ∥ > c$ ; guarding on this comparison is exactly how real code (clip_grad_norm_) avoids the $2/0$ .
$\overset{g}{^} = (0, 0, 0)$ , unchanged. Why? A zero gradient means "no update," and clipping leaves it that way.

Track (b) — the scalar gradient:

$∥ g ∥ = (- 7)^{2} = 7 = ∣ - 7 ∣$ . Why? With a single component, the L2 norm collapses to absolute value.
Test the shrink condition: $7 > 4$ is true → shrink. Why this step? Same comparison as always, applied to a 1-D vector.
$α = min (1, 4/7) = 4/7$ , and $\overset{g}{^} = - 7 \cdot \frac{4}{7} = - 4$ . Why? Scale to length $c = 4$ ; the sign stays because $α > 0$ .

Verify: (a) output is all zeros, no NaN. ✓ (b) $∣ \overset{g}{^} ∣ = ∣ - 4∣ = 4 = c$ , sign preserved. ✓

Ex 8 — real-world word problem: an RNN loss spike (D1)

Worked example Statement

You are training an RNN with BPTT. Normal gradient norms hover around $0.8$ . On step 4021 the norm suddenly reads $540$ and the loss is about to go NaN. You have clip_grad_norm_ set to $c = 5$ . By what factor does this step shrink, and does learning direction survive?

Forecast: Guess the shrink factor's order of magnitude before computing.

Is it a spike? $540 ≫ 0.8$ typical → yes, this is an exploding gradient off a cliff. Why? We only want to clip the rare spike, not the ordinary steps.
$∥ g ∥ = 540 > 5 = c$ → shrink. $α = 5/540 = \frac{1}{108} \approx 0.009259$ . Why? Same scalar formula, applied to the whole flattened parameter vector (global norm).
After clipping $∥ \overset{g}{^} ∥ = 5$ instead of $540$ , so a giant leap becomes a sane step in the same direction. Why global norm? Scaling every layer by one factor $α$ keeps their relative sizes intact.

Verify: $α \cdot 540 = \frac{5}{540} \cdot 540 = 5 = c$ . ✓ The direction is untouched (one positive scalar), the magnitude is safe, NaN averted. This is exactly why clipping is near-mandatory for RNNs/Transformers.

Ex 9 — exam twist: solve for the threshold (D2)

Recall Rapid self-test (predict, then reveal)

Do these in your head using "same arrow, shorter arrow." $g = (9, 12)$ , $c = 3$ : shrink factor and result? ::: $∥ g ∥ = 15$ , $α = 3/15 = 0.2$ , $\overset{g}{^} = (1.8, 2.4)$ , length $3$ . $g = (0, 0)$ , $c = 1$ , clip-by-norm: result? ::: $(0, 0)$ — safe branch, no division, no NaN. $g = (4, - 3)$ value-clipped at $v = 3$ : result and did direction change? ::: $(3, - 3)$ ; direction $4 : - 3 \to 1 : - 1$ , yes it changed. Clip-by-norm output length always equals what when clipping fires? ::: The threshold $c$ exactly.

Worked examples — Gradient clipping

The scenario matrix

Worked examples

Ex 1 — safe gradient, clip-by-norm (A1)

Ex 2 — exploding gradient, clip-by-norm (A2)

Ex 3 — exactly on the leash (A3)

Ex 4 — negative components survive scaling (A4)

Ex 5 — clip-by-value, mixed signs (B1)

Ex 6 — same vector, two rules, direction contrast (B2)

Ex 7 — zero gradient & scalar gradient (degenerate) (C1, C2)

Ex 8 — real-world word problem: an RNN loss spike (D1)

Ex 9 — exam twist: solve for the threshold (D2)

Connections