Foundations — Optimization — SGD, momentum, Adam — derivations

5.6.9 · D1 · Coding › Machine Learning (Aerospace Applications) › Optimization — SGD, momentum, Adam — derivations

Aap parent derivations ki ek bhi line padhne se pehle, aapko symbols padhne aane chahiye. Yeh page unhe ek ek karke zero se build karta hai — pehle plain words mein, phir ek picture, phir kyun yeh topic uske bina nahi chal sakta. Yahan kuch bhi assumed nahi hai.

0. Landscape picture (woh mental image jis par sab kuch tika hai)

Ek pahari ghati ka socho. Tumhari horizontal position tumhari choice of numbers hai (model ki "settings"). Tumhari height yeh hai ki un settings par model kitna galat hai. Neecha = behtar. Optimization = ghati ki talon mein pahunchna.

Yeh picture apne dimaag mein rakho. Neechay har ek symbol is picture ke kisi hisse ka label hai.

1. $\theta$ — parameters (zameen par tumhari position)

• Plain words: dials ka ek bag jinhein tum ghuma sakte ho.
• Picture: woh flat zameen jis par tum chalte ho (pahadon ke neechay x-y plane).
• Kyun yeh topic isko chahiye: training ka poora maqsad best $\theta$ dhundhna hai, isliye humein ek letter chahiye "ek saath saare dials" ki baat karne ke liye.

Ek akela number $\theta$ likha jaata hai; jab bahut saare ho tab bhi $\theta$ likhte hain lekin matlab hota hai ek vector — numbers ki ek ordered list, jaise coordinates $({\theta }_1,{\theta }_2,\dots )$ jo ek saath kai directions mein tumhari position dete hain.

2. $L(\theta )$ — loss (tumhari height)

• Plain words: tumhari current settings ka "badness score."
• Picture: jahan tum khade ho us ke upar seedha pahar ki height.
• Kyun yeh topic isko chahiye: height ke bina "neechay" kuch nahi hota. $L$ woh surface hai jis par hum utarna chahte hain.

Notation $L(\theta )$ ka matlab sirf yeh hai ki "$L$, $\theta$ par depend karta hai" — paon hilao ($\theta$ badalta hai), tumhari height ($L$) badlegi.

3. Subscript $t$ — time step (yeh kaun sa kadam hai)

• Plain words: ek stopwatch jo har update par ek baar tick karta hai.
• Picture: tumhare footprints jis order mein tum ne rakhe usi order mein numbered.
• Kyun yeh topic isko chahiye: har optimizer ek rule hai jo ${\theta }_t$ ko ${\theta }_{t+1}$ mein badalta hai. Subscript hume "purana → naya" saaf tarike se likhne deta hai.

4. Slope: derivative, partial derivative, aur $\nabla L$

Yeh asli cheez hai. Neechay chalne ke liye tumhe slope feel karni hogi. "Slope" ke teen levels, sabse simple se lekar is topic mein use hone wale tak.

4a. Derivative — EK direction mein slope

• Plain words: ek paon ke neechay zameen ki steepness.
• Picture: jahan tum khade ho wahan curve par ek seedha ramp (tangent line) rakha gaya — uska jhukav hi derivative hai.

"Graph dekho" ki jagah yeh tool kyun? Ek million-dimensional landscape ko tum dekh nahi sakte. Derivative ek formula hai jo slope numerically deta hai, koi aankhein nahi chahiye. Isliye calculus, na ki aankhein, training chalata hai.

4b. Partial derivative $\partial$ — doosron ko rokke slope

• Picture: khade raho, phir sirf east-west slide karo aur jhukav mapo; woh ek partial hai.
• Kyun zaroorat hai: har dial ka apna slope hota hai. Poori downhill direction jaanane ke liye humein sab chahiye.

4c. Gradient $\nabla L$ — ek arrow mein saari slopes

• Plain words: woh single arrow jo kehta hai "upar woh taraf hai, aur itna steep hai."
• Picture: zameen par ek arrow bana hua, tumhare upar pahar ki sabse steep slope ki taraf point karta hua.
• Kyun yeh topic isko chahiye: pehla update rule hi ${\theta }_{t+1}={\theta }_t-\eta \nabla L$ hai. Ise padhne ke liye jaanna zaroori hai ki $\nabla L$ "uphill arrow" hai aur minus sign ise downhill flip karta hai.

Gradient ko practice mein Backpropagation se compute kiya jaata hai aur yeh Taylor Expansion idea par tikha hai ki kisi bhi point ke paas ek curved surface ek flat ramp jaisi dikhti hai.

5. $\eta$ — learning rate (tumhara step length)

• Plain words: tumhare kadam ka size.
• Picture: zameen par rakhe step-arrow ki lambai.
• Kyun yeh topic isko chahiye: gradient ek direction deta hai; $\eta$ use ek actual move mein badalta hai. Bahut bada → tum ghati ko overshoot karte ho aur bounce karte ho; bahut chhota → tum ghisatte ho. Learning Rate Scheduling poori tarah $\eta$ ko samay ke saath tune karne ke baare mein hai.

6. $g_t$ — mini-batch se estimated gradient

• Plain words: kuch examples se downhill direction ka ek jaldi andaza.
• Picture: true gradient arrow, lekin thoda hilta hua kyunki tumne sirf thodi data dekhi.
• Kyun yeh topic isko chahiye: real datasets itne bade hain ki har step par exact slope feel nahi kar sakte. $g_t$ wahi hai jo SGD, Momentum, aur Adam actually use karte hain. Yeh unbiased hai (average par sahi) lekin jittery — aur woh jitter hi poori wajah hai baad ke optimizers ke liye.

Yahan ${\ell }_i$ ek example $i$ par loss hai; $|B|$ batch mein kitne examples hain; $\sum$ ka matlab hai "sab jod do."

7. $\beta$ aur exponential moving average — past ko smooth karna

• Plain words: ek smoothing knob. $\beta$ paas $1$ = lambi memory (smooth, react karne mein dhima); $\beta$ paas $0$ = choti memory (jittery).
• Picture: past gradients fade hote ja rahe hain, har purana ek pichhle se kamzor.

• Kyun yeh topic isko chahiye: noisy $g_t$ step-to-step bharosemand nahi hai. Bahut saare recent gradients ko average karna jitter cancel karta hai aur asli trend zahir karta hai. Yeh EMA hi Momentum ($v_t$) aur Adam ($m_t$, $v_t$) dono ka engine hai. Poori detail Exponential Moving Average mein.

"Exponential" wala hissa: recurrence ko unroll karne par, $k$ steps pehle ka gradient ${\beta }^k$ se weighted hota hai — aur ${\beta }^k$ geometrically shrink karta hai, isliye purani khabar tezi se fade hoti hai. Yahi upar wali picture dikhati hai.

8. Chhote helper symbols (taaki kuch surprise na kare)

• Kyun matter karte hain: $ϵ$ Adam ka division safe rakhta hai; $\mathbb{E}$ woh tarika hai jisse hum kehte hain "average par, noise ke upar"; dot product $\nabla L^{\top }\Delta \theta$ woh number hai jo batata hai kadam upar jaata hai ya neechay.

9. Landscape ke do khatarnak features

• Kyun yeh topic inhe chahiye: yeh do shapes hi woh villain hain jis se poora parent note ladhta hai. Noise (SGD) saddles se nikalta hai; memory (Momentum/Adam) ravines mein glide karta hai. Aur Saddle Points and Loss Landscapes mein.

Prerequisite map

Ise aise padho: upar plain symbols gradient ko feed karte hain, gradient aur step size milke update rule dete hain, batching SGD deta hai, smoothing Momentum deta hai, aur sab milke Adam dete hain.

Equipment checklist

Khud test karo — dayi taraf dhako.