Foundations — Batch, mini-batch, stochastic gradient descent

5.6.10 · D1 · Coding › Machine Learning (Aerospace Applications) › Batch, mini-batch, stochastic gradient descent

Parent note ko aaram se padhne se pehle, tumhe har us symbol ko kamana hoga jo woh tumhare saamne rakhta hai. Hum unhe ek ek karke build karte hain, har ek pichle wale par tikha hua. Yahan kuch bhi assume nahi kiya ki tumne calculus ya statistics dekha hai — hum ek picture se shuru karte hain.

1. Training data: $x_i$, $y_i$, $N$

Subscript $i$ ek name tag hai, multiplication nahi. $x_5$ ka matlab hai "5th input," bilkul waise jaise "chair #5" ek row mein ek kursi ki taraf point karta hai.

Figure dekho: har dot ek $(x_i,y_i)$ pair hai jo grid par plot kiya gaya hai. $N$ simply dots ki sankhya hai. Woh pura dots ka cloud sara data hai — woh cheez jise Batch GD ek baar mein dekhta hai.

2. Model aur uske knobs: $f_{\theta }$ aur $\theta$

$\theta$ ko radio ke dial ki tarah socho. Use ghoomane se jo suno badal jata hai; yahan, $\theta$ ko ghoomane se model kya predict karta hai badal jata hai. Learning = sahi dial position dhundna.

3. Ghalat hone ko measure karna: loss $L_i$ aur $J(\theta )$

Squared kyun? Do reasons, dono visual:

1. Squaring har error ko positive banata hai (upar ki miss aur neeche ki miss dono "bura" count hota hai).
2. Square ek smooth U-shaped bowl hai — aur ek bowl ka ek clear bottom hota hai jahan tum roll karke ja sakte ho. Figure dekho.

${\sum }_{i=1}^N$ ko zor se padho: "jodo, $i$ ke liye 1 se $N$ tak." Saamne $\frac{1}{N}$ us bade sum ko ek average mein badal deta hai. Toh $J$ hai "abhi average par, model kitna galat hai?"

4. Dhal: gradient ${\nabla }_{\theta }$ aur derivative

Ab key sawal: kisi $\theta$ par khade hote hue, $J$ landscape par konsi taraf neeche hai? "Konsi taraf aur kitni steeply ek curve jata hai" ka jawab dene ka tool derivative hai.

Figure mein tangent lines dekho: derivative us line ki tilt hai jo curve ko sirf kiss karti hai. Zyada tilt = bada number.

Yahi result hai jo parent ke Worked Example 1 mein use hota hai — ab tum jaante ho $\cdot x$ kahaan se aata hai, aur kyun koi $\frac{1}{2}$ nahi bachta.

Figure dekho: do-knob bowl par, do partial slopes (ek per axis) do components hain; unhe tip-to-tail join karo aur diagonal arrow ${\nabla }_{\theta }J$ hai — true steepest-uphill direction. Neeche jaane ke liye, hum arrow ke opposite jaate hain: $-{\nabla }_{\theta }J$.

5. Step: learning rate $\eta$ aur update arrow $\Delta \theta$

Sab kuch milake, ideal update hai: $\theta \leftarrow \theta -\eta {\nabla }_{\theta }J$ Arrow $\leftarrow$ ka matlab hai "replace with" — $\theta$ ko naya value assign karo. Yeh koi equation nahi hai; yeh ek action hai.

6. Estimates aur averages: $\hat{g}$, $\mathbb{E}$, aur $\mathrm{Var}$

Figure dekho: usi bowl par teen descent paths — wobbly wala (SGD, $B=1$), medium wala (mini-batch), smooth wala (Batch). Same destination, alag amounts of jitter, exactly jaisa ${\sigma }^2/B$ predict karta hai.

7. Bookkeeping words: epoch, batch size $B$

Woh $N/B$ hi parent ke Worked Example 3 ka poora content hai: $1000/100=10$ updates per epoch.

Yeh foundations topic ko kaise feed karte hain

Dependencies ki chain ko upar se neeche padho — har box is page ki ek foundation hai, aur tum koi link skip nahi kar sakte. Data $(x_i,y_i,N)$ define karta hai ki model $f_{\theta }$ ko kis cheez ke against judge kiya jata hai; woh judgement loss $L_i$ hai, landscape $J$ mein average kiya gaya; $J$ ki slope (derivative → gradient ${\nabla }_{\theta }J$) batati hai konsi taraf neeche hai; learning rate $\eta$ step ka size decide karta hai; aur aakhirkar estimate $\hat{g}$ (apne expectation aur variance ke saath) woh hai jo ideal step ko teen real algorithms mein badal deta hai.

Map use karo apna reading order check karne ke liye: agar upar koi bhi box abhi bhi fuzzy lagta hai, toh exactly wahan re-read karo teen descent methods par jaane se pehle. In ideas ke aage kahan jaate hain iske liye, dekho Gradient Descent, Learning Rate and Schedules, Loss Functions, Momentum and Adam, Saddle Points and Non-Convex Optimization, Bias-Variance Tradeoff, aur Backpropagation (woh machine jo actually real networks ke liye ${\nabla }_{\theta }J$ compute karta hai).

Equipment checklist

Self-test: kya tum reveal karne se pehle har jawab bol sakte ho?