Exercises — Gradient clipping

WHAT: compare $\Vert g\Vert$ to $c$. WHY: the rule modifies only when $\Vert g\Vert >c$ (strictly).

• (a) $4\le 10$ → unchanged (factor $\min (1,10/4)=1$).
• (b) $10\le 10$ → unchanged (factor $\min (1,10/10)=\min (1,1)=1$). Exactly on the leash counts as safe.
• (c) $25>10$ → shrunk by $\alpha =10/25=0.4$.

Clip-by-norm. It multiplies the whole vector by one scalar $\alpha$, so $\hat{g}=\alpha g$ points the same way. Clip-by-value clamps each coordinate separately, changing component ratios, so it can rotate the arrow. Mnemonic: Norm = Nice direction; Value = Violates direction.

(B). Backprop produces the raw gradient; clip it, then hand the clipped gradient to the optimizer. It never touches the loss or the model itself.

WHAT: norm first. $\Vert g\Vert =\sqrt{6^2+8^2}=\sqrt{36+64}=\sqrt{100}=10$. WHY: need to know if $10>5$. Since $10>5$: $\alpha =\min (1,5/10)=0.5$. WHY: Case 2 shrink factor. $\hat{g}=0.5\cdot (6,8)=(3,4)$. Check: $\Vert \hat{g}\Vert =\sqrt{9+16}=\sqrt{25}=5=c$. ✓

WHAT: clamp each coordinate into $[-5,5]$. $6\to 5$, $8\to 5$. So $\hat{g}=(5,5)$. Norm $=\sqrt{25+25}=\sqrt{50}\approx 7.07$ — note this is still larger than 5! Value-clipping does not bound the norm. Direction: original ratio $6:8=3:4$; new ratio $5:5=1:1$. Direction changed. See the figure below.

$\Vert g\Vert =\sqrt{0+25+144}=\sqrt{169}=13$. Since $13>6.5$: $\alpha =6.5/13=0.5$. $\hat{g}=0.5\cdot (0,5,12)=(0,2.5,6)$. Check: $\sqrt{0+6.25+36}=\sqrt{42.25}=6.5=c$. ✓

Unchanged means $\alpha =1$, i.e. $\min (1,3/\Vert g\Vert )=1$, which requires $3/\Vert g\Vert \ge 1$, i.e. $\Vert g\Vert \le 3$. So the norm is at most 3.

WHAT: after clipping, $\Vert \hat{g}\Vert =c=4$ (since $800>4$ it lands exactly on the leash). Step length $=\eta \Vert \hat{g}\Vert =0.1\cdot 4=0.4$. Unclipped: $\eta \Vert g\Vert =0.1\cdot 800=80$. Analysis: clipping cut the step from $80$ down to $0.4$ — a $200\times$ reduction — while keeping the same direction. That $80$-length leap is exactly the "fly off the cliff" jump clipping prevents.

WHAT: global norm $\Vert g\Vert =\sqrt{9^2+12^2}=\sqrt{81+144}=\sqrt{225}=15$. Since $15>7.5$: $\alpha =7.5/15=0.5$. Both blocks are scaled by the same $\alpha =0.5$: new norms $9\cdot 0.5=4.5$ and $12\cdot 0.5=6.0$. Check total: $\sqrt{4.5^2+6^2}=\sqrt{20.25+36}=\sqrt{56.25}=7.5=c$. ✓ WHY global: one shared $\alpha$ keeps the between-layer ratio $9:12$ intact ($4.5:6=9:12$). Clipping each layer to its own budget would destroy that ratio and change the true descent direction.

The factor $\min (1,c/\Vert g\Vert )$ is always $\le 1$: it can only shrink or leave unchanged. It has no mechanism to grow a tiny gradient, and vanishing = tiny gradients. Cure requires architecture — see Exploding and vanishing gradients, gating in Recurrent Neural Networks, and Weight initialization.

WHAT: find a $c$ above the "normal" cluster but below the spike. Normal norms sit around $0.9$–$1.5$; the spike is $40$. A choice like $c=5$ sits comfortably above the cluster. Which fire: only batch 4 ($40>5$). Batches with norms $1.2,0.9,1.5,1.1$ are all $\le 5$ → untouched. Justify: the whole point is to clip only the occasional cliff, not throttle every step. $c=5$ clips $1/5=20\%$... actually only the single true outlier fires, which is exactly the desired rare-spike behavior. Any $c$ roughly in $[2,20]$ works; too small (e.g. $c=1$) would clip batches 1,3,4 (the norm exceeds 1 there), needlessly capping normal learning.

Cause A — learning rate too high. Even a leashed step $\eta \Vert \hat{g}\Vert =\eta c$ can be too big if $\eta$ is large. Fix: lower $\eta$ or use an adaptive optimizer, see Optimizers - SGD Adam RMSProp. Cause B — clip applied in the wrong place. With Adam, clipping after the moment estimates (instead of on raw grads) can let a spike corrupt the running moments first. Fix: clip raw gradients before the optimizer's moment update (standard clip_grad_norm_ placement). (Bonus cause: pathological loss geometry — a true cliff — analyzed via Loss landscapes and cliffs; stabilize with Batch Normalization or better Weight initialization.)

Let $n=\Vert g\Vert$. Case $n>c$: $\hat{g}=(c/n)g$, so $\Vert \hat{g}\Vert =(c/n)n=c=\min (n,c)$ since $n>c$. ✓ Case $0<n\le c$: factor is $\min (1,c/n)=1$ (because $c/n\ge 1$), so $\hat{g}=g$ and $\Vert \hat{g}\Vert =n=\min (n,c)$. ✓ Case $n=0$ (zero gradient): $g=0$. The formula's $c/\Vert g\Vert =c/0$ is undefined, but any finite scaling of $0$ is $0$; implementations guard with $c/(\Vert g\Vert +\epsilon )$ or a branch, giving $\hat{g}=0$. Then $\Vert \hat{g}\Vert =0=\min (0,c)$. ✓ All cases give $\Vert \hat{g}\Vert =\min (\Vert g\Vert ,c)$. $■$

Clip-by-value clamps coordinates independently; clip-by-norm scales all coordinates by the same $\alpha$. For the two outputs to be identical, the value-clip must be a uniform scalar multiple of $g$.

• 1-D case: with a single component $g=(g_1)$, direction is trivially preserved. Value-clip gives $\mathrm{clip}(g_1,-v,v)$; norm-clip with $c=v$ gives $g_1\min (1,v/|g_1|)=\mathrm{sign}(g_1)\min (|g_1|,v)$ — identical. So in 1-D they always coincide (with $c=v$).
• Higher-D: they coincide iff no coordinate is clipped, i.e. $\Vert g\Vert \le c$ and ${\max }_i|g_i|\le v$; then both are the identity. Otherwise value-clip changes ratios while norm-clip preserves them, so they differ. Characterization: equal exactly when either (i) the gradient is 1-dimensional, or (ii) no clipping is needed at all.

$\Vert g\Vert =\sqrt{d{a}^2}=a\sqrt{d}$. If $a\sqrt{d}>c$, factor $\alpha =c/(a\sqrt{d})$, so each clipped coordinate is ${\hat{g}}_i=\frac{ca}{a\sqrt{d}}=\frac{c}{\sqrt{d}}.$ With $a=2,c=4$: ${\hat{g}}_i=4/\sqrt{d}$ (whenever $2\sqrt{d}>4$, i.e. $d>4$). Compare to a value bound $v=c=4$: a single norm-clipped coordinate $4/\sqrt{d}$ is always $\le 4=v$ for $d\ge 1$, so norm-clipping keeps every coordinate under the value bound automatically here. The unclipped coordinate $a=2$ never reaches $v=4$, so value-clip alone would never fire for this vector even as $d\to \infty$ — yet the norm $2\sqrt{d}\to \infty$. This is the sharpest illustration that value-clipping does not control the norm. Sanity at $d=4$: $\Vert g\Vert =2\cdot 2=4=c$, exactly on the leash, $\alpha =1$, unchanged. ✓