Question bank — Gradient clipping

False — it only edits the update step ($w\leftarrow w-\eta \hat{g}$). The loss $L$ and the model are untouched; you're just refusing to take an oversized step.

False — it multiplies the entire vector by a single positive scalar $\min (1,c/\Vert g\Vert )$. Scaling by one number rotates nothing; only the length shrinks.

False — clamping components independently changes their ratios. $(100,1)\to (10,1)$ turns a $100:1$ arrow into $10:1$, a genuinely different direction.

True — then $c/\Vert g\Vert \ge 1$, so $\min (1,c/\Vert g\Vert )=1$ and $\hat{g}=1\cdot g=g$. Safe gradients are never distorted.

False — the $\min$ can only pick a factor $\le 1$, so clipping shrinks large arrows and never grows small ones. Vanishing needs better init, gating, or residual paths.

True — with $\Vert g\Vert >c$, $\hat{g}=(c/\Vert g\Vert )g$ so $\Vert \hat{g}\Vert =(c/\Vert g\Vert )\Vert g\Vert =c$. It lands precisely on the leash, not below.

False — a lower $\eta$ shrinks every step uniformly, including good ones; clipping only touches the rare spikes. Clipping preserves fast learning on normal batches.

False — each component is $\le 5$, but the whole vector can be longer. $(5,5)$ has norm $\sqrt{50}\approx 7.07>5$; per-coordinate bounds don't bound the length.

False — global uses one norm over all params, keeping inter-layer proportions; per-layer rescales each block differently, distorting the overall direction.

False — it's near-mandatory for RNNs/LSTMs and helps BPTT, but also stabilizes Transformers and GANs wherever the loss has cliffs.

The error is over-clipping: a tiny $c$ clips almost every step, effectively throttling the learning rate and stalling training. Clip only occasional spikes, not the norm.

Wrong order — clipping edits the gradient, so it must run after backprop, before the optimizer step. Post-update there's no gradient left to clip.

The standard placement clips the raw gradient before Adam's moments are computed (clip_grad_norm_). Clipping the moments mixes clipping into an already-smoothed quantity.

It gives $(5,5)$ whose norm $\sqrt{50}\approx 7.07$ is not below $5$, and the direction flipped from $3:4$ to $1:1$. Clip-by-value here neither shortens enough nor preserves direction.

Then a safe gradient with $\Vert g\Vert <c$ would be amplified by $c/\Vert g\Vert >1$, injecting instability. The $\min$ is what guarantees "shrink-only".

Batch Normalization tames activation scales but doesn't cap the rare gradient cliff spike in recurrent unrolls; the two address different failure modes and often coexist.

At a cliff the local linear approximation breaks down, so the size of $g$ becomes unreliable, but which way is downhill is still roughly right — so we trust direction and cap length.

One shared factor keeps the relative gradient sizes between layers intact, preserving the true steepest-descent direction; per-layer factors would warp it.

For $\Vert g\Vert \le c$ it selects $1$ (do nothing); for $\Vert g\Vert >c$ it selects $c/\Vert g\Vert <1$ (shrink). Both the "safe" and "too big" branches fall out of a single expression.

A huge step overshoots the valley into a higher-loss region, whose even-bigger gradient overshoots further — a runaway that overflows floating point into Inf/NaN.

Backprop multiplies per-layer Jacobians; if their typical singular value exceeds $1$, the product grows like a geometric series in depth ($\sim r^n$, $r>1$), blowing up. See Exploding and vanishing gradients.

It shows what the norms actually look like, so you can set $c$ above the bulk and below the rare spikes — clipping the tail, not the body.

$\Vert g\Vert =0\le c$, so $\min (1,c/0)$ conceptually picks $1$ and $\hat{g}=0$. In code the $\min (1,\cdot )$ short-circuits so you never divide by zero; a zero gradient stays zero.

On the boundary $c/\Vert g\Vert =1$, so $\min$ returns $1$ and $g$ is unchanged. Clipping only bites strictly above the threshold.

Clip-by-norm does nothing (norm is under the leash), even though one coordinate is large. If you need a hard per-coordinate cap, that's the specific job of clip-by-value.

Each is on its own boundary, so clip-by-value leaves them all unchanged — but the resulting norm can be large ($v\sqrt{d}$ for $d$ components), which clip-by-value never controls.

No effective change: every step picks factor $1$. That's a sign $c$ may be set high (harmless) — clipping is a dormant safety leash that only activates on spikes.

The single global factor shrinks both proportionally, so the tiny layer's already-small gradient gets even smaller. This is accepted because preserving direction matters more; if it's a problem, the real fix is architecture/init, not per-layer clipping.