Adam and AdamW optimizers — Hinglish notes

3.2.5 · AI-ML › Training Deep Networks

Hume Adam ki zaroorat kyun hai?

Plain SGD har parameter ke liye same learning rate use karta hai: $θ \leftarrow θ - η g_{t}, g_{t} = \nabla_{θ} L$

Problem 1 — alag-alag scales. Deep nets mein, kuch parameters ko bahut bade gradients milte hain, doosron ko bahut chhote. Ek global $η$ kuch ke liye bahut bada hai, kuch ke liye bahut chhota.

Problem 2 — noisy stochastic gradients. Har minibatch ek noisy $g_{t}$ deta hai. Hum chahte hain ki noise average out ho jaye (momentum) aur saath hi normalize ho iske hisaab se ki har gradient kitna bada/variable hai.

Adam dono ka jawab deta hai, har parameter ke liye do exponential moving averages (EMAs) track karke.

Adam ko first principles se derive karna

Step 1 — First moment (momentum): direction ko smooth karo

Hum gradient ka EMA lete hain: $m_{t} = β_{1} m_{t - 1} + (1 - β_{1}) g_{t}$ Kyun? $m_{t}$ , $E [g_{t}]$ — yaani average gradient — ko estimate karta hai. Yeh noise cancel karta hai aur consistent directions ko aage move karta rehta hai.

Step 2 — Second moment: magnitude ko smooth karo

Hum squared gradient (elementwise) ka EMA lete hain: $v_{t} = β_{2} v_{t - 1} + (1 - β_{2}) g_{t}^{2}$ Kyun? $v_{t}$ , $E [g_{t}^{2}]$ — har coordinate mein gradient ki size/variance — ko estimate karta hai. Bada $v_{t}$ ⇒ woh direction steep/noisy hai ⇒ wahan chhote steps lo.

Step 3 — Bias correction (yeh hai clever wala hissa)

Kyunki $m_{0} = v_{0} = 0$ hai, early EMAs zero ki taraf biased hote hain. Chaliye correction ko prove karte hain. Maano $g$ roughly stationary hai, mean $\overset{g}{ˉ}$ ke saath. Unrolling karke: $m_{t} = (1 - β_{1}) \sum_{i = 1}^{t} β_{1}^{t - i} g_{i} .$ Expectations lete hain $E [g_{i}] = \overset{g}{ˉ}$ ke saath: $E [m_{t}] = \overset{g}{ˉ} (1 - β_{1}) \sum_{i = 1}^{t} β_{1}^{t - i} = \overset{g}{ˉ} (1 - β_{1}) \cdot \frac{1 - β _{1}^{t}}{1 - β _{1}} = \overset{g}{ˉ} (1 - β_{1}^{t}) .$ Toh $E [m_{t}] = \overset{g}{ˉ} (1 - β_{1}^{t})$ factor $(1 - β_{1}^{t})$ se bahut chhota hai. Ise divide out karo: $\overset{m}{^}_{t} = \frac{m _{t}}{1 - β _{1}^{t}}, \overset{v}{^}_{t} = \frac{v _{t}}{1 - β _{2}^{t}} .$ Ab $E [\overset{m}{^}_{t}] \approx \overset{g}{ˉ}$ — unbiased. Jab $t \to \infty$ , $β_{1}^{t} \to 0$ aur correction khatam ho jaata hai.

Step 4 — Update

$θ_{t} = θ_{t - 1} - η \frac{m ^ _{t}}{v ^ _{t} + ϵ}$

AdamW — decoupled weight decay

Adam-with-L2 (naive): decay ko gradient mein fold karo $g_{t} \leftarrow \nabla L + λ θ_{t - 1} \Rightarrow decay gets scaled by 1/ \overset{v}{^}_{t} . ❌$

AdamW: decay ko adaptive machinery ke bahar rakho $θ_{t} = θ_{t - 1} - η (\frac{m ^ _{t}}{v ^ _{t} + ϵ} + λ θ_{t - 1})$

Worked examples

Common mistakes (Steel-manned)

Recall

Recall Active recall — answers cover karo

Adam har parameter ke liye kaunse do statistics track karta hai? ⇒ Gradient ka EMA ( $m_{t}$ ) aur squared gradient ka EMA ( $v_{t}$ ).
$\overset{v}{^}_{t}$ se divide kyun karte hain? ⇒ Har coordinate ko uski gradient magnitude se normalize karne ke liye → adaptive per-parameter learning rate.
Bias-correct kyun karte hain? ⇒ EMAs 0 par initialize hote hain, factor $(1 - β^{t})$ se biased low hain; ussse divide karne par yeh remove ho jaata hai.
Constant gradient par steady-state Adam step kya hai? ⇒ $η sign (g)$ .
AdamW exactly kya change karta hai? ⇒ weight decay $η λ θ$ ko $\overset{v}{^}$ normalization ke bahar (decoupled) apply karta hai.

Recall Feynman: 12-saal ke bachche ko explain karo

Socho tum fog mein pahadi utaar rhe ho. Momentum ( $m$ ) matlab hai yaad rakhna ki tum kis taraf chal rahe ho taaki zig-zag na ho. Magnitude tracker ( $v$ ) notice karta hai kaunsi directions steep aur slippery hain, toh wahan tiny careful steps lo aur gentle slopes par bade confident steps lo. Bias correction isliye hai kyunki bilkul shuru mein tumhare paas koi memory nahi hoti, toh tum apne pehle guesses ko multiply up karte ho taaki woh fair lagein. AdamW ek rule add karta hai: "har step mein, apni saari cheezein thodi si zero ki taraf shrink karo, equally" — taaki kuch bhi bahut bada aur messy na ho jaye. Adam = smart footwork; AdamW = smart footwork plus tidy-up.

Flashcards

Adam first moment update formula

m_{t} = β_{1} m_{t - 1} + (1 - β_{1}) g_{t}

Adam second moment update formula

v_{t} = β_{2} v_{t - 1} + (1 - β_{2}) g_{t}^{2}

Adam bias-corrected estimates

\overset{m}{^}_{t} = m_{t} / (1 - β_{1}^{t})

\overset{v}{^}_{t} = v_{t} / (1 - β_{2}^{t})

Adam parameter update

θ_{t} = θ_{t - 1} - η \overset{m}{^}_{t} / (\overset{v}{^}_{t} + ϵ)

Bias correction kyun zaroori hai

EMAs 0 par initialize hote hain,

(1 - β^{t})

factor se zero ki taraf biased hain; ussse divide karne par estimate unbiased ho jaata hai.

Default Adam hyperparameters

β_{1} = 0.9, β_{2} = 0.999, ϵ = 1 0^{- 8}

\overset{m}{^} / \overset{v}{^}

ka interpretation

yeh ek signal-to-noise ratio hai; consistent gradients → step ≈ η, noisy gradients → chhota step.

Constant gradient g par steady-state Adam step

η sign (g)

— magnitude

∣ g ∣

se independent.

AdamW, Adam+L2 se kya change karta hai

weight decay

η λ θ

\overset{v}{^}

normalization se bahar (decoupled) apply karta hai.

Adam+L2 decay galat kyun hai

λ θ

term

\overset{v}{^}

se divide ho jaata hai, toh decay uniform ki jagah gradient-dependent ban jaati hai.

AdamW update equation

θ_{t} = θ_{t - 1} - η \overset{m}{^}_{t} / (\overset{v}{^}_{t} + ϵ) - η λ θ_{t - 1}

ϵ

ka role /0 avoid karne ke alawa

max effective learning rate (

\leq η / ϵ

) cap karta hai jab

\overset{v}{^}

tiny ho.

Connections

Stochastic Gradient Descent — woh baseline jise Adam generalize karta hai.
Momentum and Nesterov Acceleration — first-moment term ka source.
RMSProp — second-moment normalization ka source.
Exponential Moving Average — core smoothing tool aur iska bias.
Weight Decay and L2 Regularization — jo AdamW decouple karta hai.
Learning Rate Schedules — warmup naturally Adam ki early instability ke saath pair hota hai.
Bias-Variance in Gradient Estimation — kyun hum gradients smooth karte hain.