Gradient clipping

WHAT is gradient clipping?

WHY do we need it? (First principles)

Consider a deep network as a composition of layers. Backprop multiplies Jacobians together:

$\frac{\partial L}{\partial h_0} = \frac{\partial L}{\partial h_n}\prod_{k=1}^{n}\frac{\partial h_k}{\partial h_{k-1}}$

If the typical singular value of each factor $\frac{\partial h_k}{\partial h_{k-1}}$ is $> 1$ , this product grows geometrically with depth $n$ (like $2^n$ ). This is the exploding gradient problem.

The key insight: near a "cliff" in the loss landscape (common in recurrent models), the gradient's magnitude becomes unreliable, but its direction is still roughly correct. So we keep the direction and just rescale the length.

HOW: Clip-by-norm (the standard one)

Formula Clip-by-global-norm — derived

Goal: limit the total step length. Let $g$ be the full gradient (all params flattened into one vector) with L2 norm $\|g\|$ . Pick a threshold $c$ .

Case 1 — gradient is already safe ( $\|g\| \le c$ ): do nothing, $\hat g = g$ .

Case 2 — gradient too big ( $\|g\| > c$ ): we want a new vector $\hat g$ that

points in the same direction as $g$ , so $\hat g = \alpha g$ for some scalar $\alpha>0$ , and
has norm exactly $c$ , so $\|\hat g\| = \alpha\|g\| = c \Rightarrow \alpha = \dfrac{c}{\|g\|}$ .

Combining both cases: $\boxed{\;\hat g = g \cdot \min\!\left(1,\ \frac{c}{\|g\|}\right)\;}$

Why the $\min$ ? When $\|g\|\le c$ , we get $c/\|g\|\ge 1$ , so $\min$ picks $1$ (no change). When $\|g\|>c$ , $c/\|g\|<1$ , so we shrink. One formula, both cases. Clean.

HOW: Clip-by-value

Worked examples

Practical notes

Typical threshold $c$ : 1.0 to 5.0 (task-dependent; tune it). Watch the unclipped norm histogram to pick it.
Apply clipping after backprop, before optimizer step.
With Adam, clip raw gradients (before the moment estimates), which is the standard clip_grad_norm_ placement.
Almost mandatory for RNNs/LSTMs/Transformers; also stabilizes GANs.

Forecast-then-Verify

Recall Predict before reading the answer

Q: $g=(0,5,12)$ , clip-by-norm with $c=6.5$ . What is $\hat g$ ? Forecast, then check: $\|g\|=\sqrt{0+25+144}=\sqrt{169}=13$ . $\alpha=\min(1,6.5/13)=0.5$ . $\hat g=(0, 2.5, 6)$ , norm $=\sqrt{0+6.25+36}=\sqrt{42.25}=6.5$ . ✓

Feynman: explain to a 12-year-old

Recall Explain simply (hidden)

Imagine you're walking down a hill blindfolded, taking steps in the steepest downhill direction. Usually your steps are normal-sized. But sometimes the ground is super steep and you take a GIANT leap — so big you fly right past the bottom and land higher up on the other side! Gradient clipping is like putting a rule: "no matter how steep it looks, never take a step longer than 2 meters." You still walk in the correct downhill direction — you just refuse to make crazy-big jumps. That keeps you from launching off the mountain.

Connections

Flashcards

What problem does gradient clipping solve?

Exploding gradients — it caps gradient magnitude so update steps don't overshoot and diverge (NaNs).

Clip-by-norm formula?

\hat g = g\cdot\min(1, c/\|g\|)

where

c

is the threshold and

\|g\|

the L2 norm.

Why the

\min

in the clip-by-norm formula?

It leaves safe gradients (

\|g\|\le c

) unchanged (factor 1) and shrinks only oversized ones (factor

c/\|g\|<1

Does clip-by-norm change the gradient direction?

No — it rescales the whole vector by one scalar, preserving direction; only the length changes.

Does clip-by-value preserve direction?

Not necessarily — clamping each component independently can change component ratios and thus the direction.

Can gradient clipping fix vanishing gradients?

No — it only shrinks large gradients; vanishing needs architectural fixes (gating, residuals, init).

Where in the pipeline is clipping applied?

After backprop computes gradients, before the optimizer update step.

g=(3,4)

c=2

, clip-by-norm result?

\|g\|=5

\alpha=0.4

\hat g=(1.2,1.6)

with norm exactly 2.

Symptom of exploding gradients during training?

Loss suddenly spikes or becomes NaN/Inf; gradient norms blow up.

Risk of setting the clip threshold too small?

Almost every step gets clipped, effectively throttling learning and slowing convergence.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, deep networks — khaas kar ke RNN aur Transformers — train karte waqt ek dikkat aati hai jise exploding gradients kehte hain. Backprop mein har layer ke gradients ek dusre se multiply hote jaate hain, aur agar values 1 se badi hui to depth ke saath ye geometrically badh jaati hain (jaise $2^n$ ). Result: gradient ki norm kabhi kabhi $10^6$ tak pahunch jaati hai, aur jab tum $w \leftarrow w - \eta g$ karte ho to step itna bada ho jata hai ki tum valley ko cross kar ke aur upar chale jaate ho. Loss NaN ho jaata hai. Bas yahi rokne ke liye gradient clipping aata hai.

Idea bahut simple hai: gradient ki direction sahi hoti hai, sirf uski length (magnitude) pagal ho jaati hai. To hum ek threshold $c$ set karte hain. Agar $\|g\| \le c$ hai, kuch mat karo. Agar $\|g\| > c$ hai, to poore vector ko ek hi scalar $c/\|g\|$ se multiply kar do — isse direction same rehti hai par norm exactly $c$ ho jaati hai. Ek line ka formula: $\hat g = g \cdot \min(1, c/\|g\|)$ . Yeh hai clip-by-norm, aur yahi standard hai.

Ek doosra tareeka hai clip-by-value, jisme har component ko alag se $[-v, v]$ mein clamp karte ho. Lekin iska ek chhupa hua nuksaan hai — yeh gradient ki direction badal deta hai. Jaise $(100,1)$ ko $v=10$ pe clip karo to $(10,1)$ mil jata hai, ratio $100{:}1$ se $10{:}1$ ho gaya. Isliye jab direction preserve karni ho, clip-by-norm use karo.

Yaad rakhna do baatein: (1) clipping sirf exploding gradients theek karti hai, vanishing nahi — vanishing ke liye LSTM gates ya residual connections chahiye. (2) Threshold $c$ ko bahut chhota mat rakho, warna har step clip hoga aur learning slow pad jaayegi. Typical value 1 se 5 ke beech, aur backprop ke baad, optimizer step se pehle apply karo. Bas — "same arrow, shorter arrow" yaad rakho!

WHAT is gradient clipping?

WHY do we need it? (First principles)

HOW: Clip-by-norm (the standard one)

HOW: Clip-by-value

Worked examples

Practical notes

Forecast-then-Verify

Feynman: explain to a 12-year-old

Connections

Flashcards

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Training Deep Networks

Connections