3.2.14
Intuition Ek sentence mein ideaJab deep networks train karte hain (khaaskar RNNs/Transformers), gradients kabhi kabhi explode hokar itni badi values le lete hain ki weights ek cliff se ud jaate hain. Gradient clipping ek safety leash hai: agar gradient bahut bada ho jaaye, to update step lene se pehle hum use shrink back kar lete hain ek safe size mein — taaki learning ki direction preserve rahe lekin magnitude sane rahe.
Definition Gradient clippingEk technique jo gradient ki size ko cap karti hai usse optimizer update mein use karne se pehle. Do main flavors hain:
Clip-by-norm : poore gradient vector ko rescale karo agar uska norm ek threshold se zyada ho.Clip-by-value : har individual component ko [ − v , + v ] nahi badalta — sirf update step ko.
Ek deep network ko layers ki composition socho. Backprop Jacobians ko multiply karta hai ek saath:
∂ h 0 ∂ L = ∂ h n ∂ L ∏ k = 1 n ∂ h k − 1 ∂ h k
Agar har factor ∂ h k − 1 ∂ h k > 1 n geometrically grow karta hai (jaise 2 n exploding gradient problem hai.
Intuition Exploding gradients training ko kyun barbad karte hainGradient descent step hai w ← w − η g g 1 0 6 η g divergence . Loss NaN ho jaata hai.
Asli insight: loss landscape mein ek "cliff" ke paas (jo recurrent models mein common hai), gradient ki magnitude unreliable ho jaati hai, lekin uski direction phir bhi roughly sahi hoti hai. To hum direction rakhte hain aur bas length ko rescale karte hain.
min ∥ g ∥ ≤ c c /∥ g ∥ ≥ 1 min 1 ∥ g ∥ > c c /∥ g ∥ < 1
Global norm kyun (per-layer nahi)?
Saare parameters par EK norm use karne se layers ke beech relative sizes intact rehti hain — tum sab ko same factor se rescale karte ho. Har layer ko alag clip karna true gradient direction ko distort kar deta.
Common mistake Clip-by-valuedirection badal sakta hai!
Kyun sahi lagta hai: "Bas har number ko cap karo, sabse simple cheez." Kyun subtly galat hai: agar g = ( 100 , 1 ) v = 10 ( 10 , 1 ) 100 : 1 10 : 1 alag direction mein step le rahe ho true steepest descent se. Fix: clip-by-norm prefer karo jab direction preserve karna ho; clip-by-value tab use karo jab tumhe specifically per-coordinate hard bound chahiye.
Worked example Example 1 — clip-by-norm, gradient bahut badaGradient g = ( 3 , 4 ) c = 2
Norm compute karo: ∥ g ∥ = 3 2 + 4 2 = 25 = 5 Kyun? Jaanne ke liye ki kya hum c
Kyunki 5 > 2 α = min ( 1 , 2/5 ) = 0.4 Kyun? Rule Case 2.
g ^ = 0.4 ⋅ ( 3 , 4 ) = ( 1.2 , 1.6 ) Kyun? Same direction, shrunk.Check karo: ∥ g ^ ∥ = 1. 2 2 + 1. 6 2 = 1.44 + 2.56 = 4 = 2 = c
Worked example Example 2 — clip-by-norm, gradient pehle se safeg = ( 0.3 , 0.4 ) c = 2 ∥ g ∥ = 0.5 ≤ 2 α = min ( 1 , 2/0.5 ) = min ( 1 , 4 ) = 1 Kyun? Koi danger nahi, ek acche gradient ko distort mat karo.
Worked example Example 3 — clip-by-value vs clip-by-norm ek hi vector parg = ( 6 , 8 ) c = v = 5
By value: g ^ = ( 5 , 5 ) 6 : 8 = 3 : 4 1 : 1 Direction badal gayi. Norm = 50 ≈ 7.07 ≤ 5 By norm: ∥ g ∥ = 10 α = 5/10 = 0.5 g ^ = ( 3 , 4 ) 5 Contrast kyun matter karta hai: dikhata hai ki clip-by-norm geometry respect karta hai; clip-by-value nahi karta.
Typical threshold c 1.0 to 5.0 (task-dependent; tune karo). Unclipped norm histogram dekho use pick karne ke liye.
Clipping backprop ke baad, optimizer step se pehle apply karo.
Adam ke saath, raw gradients clip karo (moment estimates se pehle), jo standard clip_grad_norm_ placement hai.
RNNs/LSTMs/Transformers ke liye almost mandatory hai; GANs ko bhi stabilize karta hai.
Common mistake "Clipping vanishing gradients bhi fix karta hai."Kyun sahi lagta hai: "Yeh gradient fix hai, to gradient problems fix karta hai." Kyun galat hai: clipping sirf bade gradients ko shrink karta hai — chote ones ko kabhi grow nahi karta. Vanishing gradients ke liye architecture fixes chahiye (LSTM gates, residual connections, better init/normalization). Fix: exploding ke liye clipping use karo, vanishing ke liye skip-connections/gating use karo.
c Kyun sahi lagta hai: "Chhota = safer." Kyun galat hai: bahut chhota c almost har step ko clip karta hai, effectively large-signal batches ke liye learning rate cap kar deta hai aur learning slow/stall ho jaati hai. Fix: sirf rare spikes clip karo — norm ko c
Recall Jawab padhne se pehle predict karoQ: g = ( 0 , 5 , 12 ) c = 6.5 g ^ ∥ g ∥ = 0 + 25 + 144 = 169 = 13 α = min ( 1 , 6.5/13 ) = 0.5 g ^ = ( 0 , 2.5 , 6 ) = 0 + 6.25 + 36 = 42.25 = 6.5
Recall Simply explain karo (hidden)Socho tum ek pahaad se aankhein bandh karke utar rahe ho, sabse steep downhill direction mein kadam rakhte hue. Zyaadaatar baar tumhare kadam normal size ke hote hain. Lekin kabhi kabhi zameen bahut zyada steep hoti hai aur tum ek BAHUT BADA leap lete ho — itna bada ki tum sidha bottom se nikalkaar doosri taraf upar ja girte ho! Gradient clipping ek aisi rule lagne jaisi hai: "chahe kitna bhi steep lage, kabhi 2 meter se lamba step mat lena." Tum phir bhi sahi downhill direction mein chalte ho — bas crazy-bade jumps lene se mana kar dete ho. Isse tum pahaad se launch hone se bache rehte ho.
"Same arrow, shorter arrow." Clip-by-norm arrow ki direction rakhta hai, sirf uski length trim karta hai. (Aur: N orm = N ice direction; V alue = V iolates direction.)
Gradient clipping kaunsi problem solve karta hai? Exploding gradients — yeh gradient magnitude ko cap karta hai taaki update steps overshoot na karein aur diverge na hon (NaNs).
Clip-by-norm formula kya hai? g ^ = g ⋅ min ( 1 , c /∥ g ∥ ) c ∥ g ∥
Clip-by-norm formula mein min Yeh safe gradients (∥ g ∥ ≤ c c /∥ g ∥ < 1
Kya clip-by-norm gradient direction change karta hai? Nahi — yeh poore vector ko ek scalar se rescale karta hai, direction preserve karta hai; sirf length change hoti hai.
Kya clip-by-value direction preserve karta hai? Necessarily nahi — har component ko independently clamp karna component ratios aur is tarah direction badal sakta hai.
Kya gradient clipping vanishing gradients fix kar sakta hai? Nahi — yeh sirf bade gradients shrink karta hai; vanishing ke liye architectural fixes chahiye (gating, residuals, init).
Pipeline mein clipping kahan apply hoti hai? Backprop ke gradients compute karne ke baad, optimizer update step se pehle.
g = ( 3 , 4 ) c = 2 ∥ g ∥ = 5 α = 0.4 g ^ = ( 1.2 , 1.6 )
Training mein exploding gradients ka symptom kya hai? Loss achanak spike karta hai ya NaN/Inf ho jaata hai; gradient norms blow up ho jaate hain.
Clip threshold bahut chhota set karne ka risk kya hai? Almost har step clip ho jaata hai, effectively learning ko throttle karta hai aur convergence slow kar deta hai.
Backprop multiplies Jacobians
