Worked examples — Batch, mini-batch, stochastic gradient descent

5.6.10 · D3 · Coding › Machine Learning (Aerospace Applications) › Batch, mini-batch, stochastic gradient descent

Yeh page parent topic ka drill floor hai. Yahan theory dobara explain nahi ki gayi — hum har case ko cover karte hain jo yeh topic throw kar sakta hai, ek worked example per cell. Agar koi symbol unfamiliar lage, parent par wapas jaao, phir return karo.

Shuru karne se pehle, ek one-line reminder us machine ka jo hum chala rahe hain:

---

The scenario matrix

Is topic ka har problem in cells mein se kisi ek mein aata hai. Har row ek aisi cheez hai jo vary ho sakti hai; neeche ke examples us cell ke saath labelled hain jo woh cover karta hai.

#	Case class	Kya mushkil banata hai	Example
C1	Gradient ki sign — positive $\hat{g}$	step $\theta$ mein neeche jaana chahiye, upar nahi	Ex 1
C2	Gradient ki sign — negative $\hat{g}$	step $\theta$ mein upar jaata hai	Ex 2
C3	Zero gradient (degenerate)	already ek flat spot par — koi movement nahi	Ex 3
C4	Batch vs SGD same data par	noise vs exactness numerically dikhao	Ex 4
C5	Variance $1/B$ ki tarah shrink hoti hai	noise-reduction number prove karo	Ex 5
C6	Epoch bookkeeping, $N$ divisible by $B$ nahi	leftover last batch	Ex 6
C7	Limiting behaviour — $\eta$ bahut bada	divergence / overshoot	Ex 7
C8	Real-world word problem (aerospace)	wall-clock ko method choice mein translate karo	Ex 8
C9	Exam twist — linear-scaling rule	$\eta$ ko $B$ ke saath sahi se scale karo	Ex 9
C10	Saddle se escape (noise kyun help karta hai)	zero gradient jo noise break karta hai	Ex 10

Numeric examples ke liye running toy model sabse simple cheez hai jiske paas phir bhi gradient ho:

$L_i=\frac{1}{2}(\theta x_i-y_i{)}^2,{\nabla }_{\theta }{L}_i=(\theta x_i-y_i)x_i.$

Beech wala formula chain rule se aata hai: $\frac{1}{2}(\theta x_i-y_i{)}^2$ ko differentiate karo aur inner $x_i$ bahar aa jaata hai. Hum ise har example mein use karte hain. Throughout, hum subscript $i$ rakhte hain ${\nabla }_{\theta }{L}_i$ par taaki yeh hamesha clear rahe ki hum ek sample ki baat kar rahe hain.

---

C1 — Positive gradient: step neeche jaana chahiye $\theta$ mein

Forecast: hum target ke upar se shuru kar rahe hain. Guess — ek step ke baad $\theta$ increase hoga ya decrease?

1. Gradient compute karo. ${\nabla }_{\theta }{L}_1=(\theta x_1-y_1)x_1=(3\cdot 2-4)\cdot 2=(2)\cdot 2=+4$. Kyunki yeh SGD hai ($B=1$), estimate is exact per-sample gradient ke equal hai: $\hat{g}={\nabla }_{\theta }{L}_1=+4$. Yeh step kyun? Yahi ek cheez hai jo update rule ko chahiye; iska sign direction decide karta hai.
2. Sign padho. $\hat{g}=+4>0$. Ek positive gradient ka matlab hai "loss $\theta$ badhne par badhti hai," toh hum $\theta$ ko neeche move karna chahiye. Yeh step kyun? Rule $\theta -\eta \hat{g}$ ek positive number subtract karta hai → $\theta$ shrink hota hai. $\hat{g}$ ki sign = direction, automatically.
3. Update. $\theta \leftarrow 3-0.1(4)=2.6$. Yeh step kyun? Humne general rule liya aur $\hat{g}$ plug in kiya.

Verify: humne predict kiya tha ki $\theta$ ${\theta }^{\ast }=2$ ki taraf drop karega. Target se distance $|3-2|=1$ se ghatkar $|2.6-2|=0.6$ ho gaya, toh hum kareeб aaye. ✓ Positive gradient ⇒ downward step. ✓

---

C2 — Negative gradient: step upar jaata hai $\theta$ mein

Forecast: is baar target se neeche. Upar ya neeche?

1. Gradient: ${\nabla }_{\theta }{L}_1=(0\cdot 2-4)\cdot 2=-8$. Phir se SGD, toh $\hat{g}={\nabla }_{\theta }{L}_1=-8$. Yeh step kyun? Same formula; sirf $\theta$ change hua.
2. Sign: $\hat{g}=-8<0$. Ek negative number subtract karna add karta hai, toh $\theta$ badhta hai. Yeh step kyun? $-\eta \hat{g}=-0.1(-8)=+0.8$ — sign flip poora mechanism hai.
3. Update: $\theta \leftarrow 0-0.1(-8)=0.8$.

Verify: ${\theta }^{\ast }=2$ se distance $|0-2|=2\to |0.8-2|=1.2$ ho gaya. Kareeб. ✓ Negative gradient ⇒ upward step. ✓ C1 aur C2 milkar dikhate hain ki $\hat{g}$ ki sign hamesha humein minimum ki taraf steer karti hai, dono sides se.

---

C3 — Zero gradient: degenerate "already there" case

Forecast: hum ${\theta }^{\ast }$ par hain. Ek achha algorithm kya karna chahiye? (Hint: kuch nahi.)

1. Gradient: ${\nabla }_{\theta }{L}_1=(2\cdot 2-4)\cdot 2=(0)\cdot 2=0$, toh $\hat{g}=0$. Yeh step kyun? Minimum par prediction $\theta x_1=4$ target $y_1=4$ ke equal hai; residual $(\theta x_1-y_1)$ zero hai, poore gradient ko kill karta hai.
2. Update: $\theta \leftarrow 2-0.1(0)=2$. Koi movement nahi. Yeh step kyun? $\hat{g}=0\Rightarrow$ step size $0$ hai chahe $\eta$ kuch bhi ho.

Verify: loss $L_1=\frac{1}{2}(4-4{)}^2=0$, already minimal; algorithm sahi se move karna band kar deta hai. ✓ Yeh update ka fixed point hai: $\theta$ stable hai iff $\hat{g}=0$. (Dhyan do: yahi condition saddle points par bhi hold karti hai — C10 dekho.)

---

C4 — Identical data par Batch vs SGD (noise vs exactness)

Forecast: kaun sa step bada hoga — Batch, ya worst-case single-sample SGD step?

1. Per-sample gradients: ${\nabla }_{\theta }{L}_1=(0-4)\cdot 2=-8$, ${\nabla }_{\theta }{L}_2=(0-1)\cdot 1=-1$. Yeh step kyun? Humein har ek ki zaroorat hai dono average (Batch) aur pick (SGD) karne ke liye.
2. Batch estimate: ${\hat{g}}_{\text{batch}}=\frac{1}{2}(-8-1)=-4.5$ — yahan $\hat{g}\ne$ koi bhi single ${\nabla }_{\theta }{L}_i$; yeh $N=2$ par unka average hai, exact ${\nabla }_{\theta }J$. Update: $\theta \leftarrow 0-0.1(-4.5)=0.45$. Yeh step kyun? Batch saare $N$ samples par mean use karta hai.
3. SGD estimate, sample 1: $\hat{g}={\nabla }_{\theta }{L}_1=-8$ → $\theta \leftarrow 0.8$. SGD, sample 2: $\hat{g}={\nabla }_{\theta }{L}_2=-1$ → $\theta \leftarrow 0.1$. Yeh step kyun? SGD ($B=1$) ek random sample pick karta hai, toh $\hat{g}$ us sample ke exact gradient ke equal hota hai — kismat par depend karke $0.8$ ya $0.1$ produce karta hai. Woh kismat hi "noise" hai.

Verify: do SGD outcomes ka average $=\frac{1}{2}(0.8+0.1)=0.45=$ Batch outcome. ✓ Yeh "SGD unbiased hai" ko numerically dikhaya gaya hai: on average SGD Batch ke equal hai, lekin koi bhi single SGD step (0.8 ya 0.1) uske around scatter karta hai. Woh scatter noise hai.

---

C5 — Variance sach mein $1/B$ ki tarah shrink hoti hai

Forecast: $B=1$ se $B=9$ jaane par — noise $9\times$ drop hoti hai ya $3\times$?

1. Formula apply karo $\mathrm{Var}[\hat{g}]={\sigma }^2/B$ (dekho Bias-Variance Tradeoff). Yeh step kyun? $\hat{g}$ $B$ i.i.d. gradients ka average hai; $B$ terms ke mean ka variance ${\sigma }^2/B$ hota hai.
2. Plug in: $B=1\Rightarrow 36$; $B=4\Rightarrow 9$; $B=9\Rightarrow 4$.
3. Catch — standard deviation. Jo noise feel hoti hai woh standard deviation $\sqrt{\mathrm{Var}}$ hai: $6,3,2$. $B=1\to 9$ se std $6\to 2$ drop hoti hai, yani factor of $3=\sqrt{9}$, nahi $9$. Yeh step kyun? Yeh parent ka "diminishing returns" fact hai: noise $1/\sqrt{B}$ ki tarah girta hai, toh $B$ double karna kami kam help karta hai.

Verify: $36/1=36$, $36/4=9$, $36/9=4$; aur $\sqrt{36},\sqrt{9},\sqrt{4}=6,3,2$. Ratio $6/2=3=\sqrt{9/1}$. ✓ Cell C5 confirmed.

---

C6 — Epoch bookkeeping jab $N$ divisible by $B$ nahi hai

Forecast: $1000/300=3.33$ — lekin aadha update ho nahi sakta. Kis taraf round karo?

1. Full batches: $\lfloor 1000/300\rfloor =3$ full batches of 300 = 900 samples. Yeh step kyun? Har update exactly $B$ fresh samples consume karta hai.
2. Leftover: $1000-900=100$ samples bache hain — ek partial last batch. Yeh step kyun? Epoch finish karne ke liye humein saare $N$ data ek baar cover karne honge, toh remainder ko bhi apna update milta hai.
3. Total updates: $\lceil 1000/300\rceil =4$ updates per epoch (3 full + 1 partial). Yeh step kyun? Ceiling, kyunki remainder phir bhi ek update trigger karta hai.

Verify: $3\cdot 300+100=1000$ ✓, aur $\lceil 1000/300\rceil =4$ ✓. Contrast: Batch GD = 1 update/epoch, pure SGD = 1000 updates/epoch. Mini-batch 4 ke saath beech mein baitha hai.

---

C7 — Limiting behaviour: learning rate bahut bada ⇒ divergence

Forecast: bada $\eta$ = bade steps. Kya $\theta$ $2$ par settle hoga, ya hamesha ke liye uss se aage nikal jaayega?

1. Step 1 gradient: $\hat{g}=(0\cdot 2-4)\cdot 2=-8$. Update: $\theta \leftarrow 0-0.5(-8)=4$. Yeh step kyun? Humne target ${\theta }^{\ast }=2$ overshoot kiya aur doosri side par land kiya ($4>2$).
2. Step 2 gradient $\theta =4$ par: $\hat{g}=(4\cdot 2-4)\cdot 2=(4)\cdot 2=+8$. Update: $\theta \leftarrow 4-0.5(8)=0$. Yeh step kyun? Ab positive gradient wapas push karta hai — lekin same magnitude se, seedha $0$ par wapas.
3. Diagnose: $\theta$ ping-pong karta hai $0\to 4\to 0\to 4\dots$ Yeh kabhi converge nahi karta; loss hamesha high rehta hai.

Verify: $\theta =0$ par, $L_1=\frac{1}{2}(0-4{)}^2=8$; $\theta =4$ par, $L_1=\frac{1}{2}(8-4{)}^2=8$ — equal, koi progress nahi. ✓ Is quadratic ke liye stability boundary $\eta <2/x_1^2=2/4=0.5$ hai; hum exactly uس par baitha aur ek perpetual oscillation mili. Koi bhi bada aur yeh blow up hota. Isliye schedules aur Momentum and Adam jaisi methods exist karti hain.

---

C8 — Real-world aerospace word problem

Forecast: teeno us hi 2M samples ko per epoch touch karte hain. Toh kaun sa us pass ke andar sabse zyada update karta hai?

1. Compute cost per epoch sabke liye same hai: saare $N$ = 2M samples ek baar touch karna $=2,000,000\times 10^{-6}=2$ seconds. 30 s ke andar bilkul. Achha — differentiator update count hai, total compute nahi. Yeh step kyun? Per epoch evaluate kiye gaye gradients ki sankhya $N$ hai har method ke liye; sirf aap unhe updates mein kaise group karte ho (via $B$) woh change hota hai.
2. Batch GD: saare 2M ko 1 update mein group karta hai. Tum $\theta$ ko 2 seconds ke compute mein ek baar improve karte ho. Yeh step kyun? $B=N$ ⇒ $N/B=1$.
3. SGD: $N/B=2,000,000$ tiny updates — extreme lekin bahut jittery aur cache-unfriendly.
4. Mini-batch $B=256$: $\lceil 2,000,000/256\rceil =7813$ updates per epoch — bahut saare achhe steps, GPU-friendly. Yeh practical winner hai. Yeh step kyun? $2,000,000/256=7812.5$, ceiling $7813$ (last batch mein $128$ samples hain).

Verify: $2,000,000\times 10^{-6}=2$ s ✓ (< 30 s, toh sab feasible hain compute par). $7812\cdot 256=1,999,872$, remainder $128$, toh $\lceil \cdot \rceil =7813$ ✓. Batch=1, SGD=2,000,000, mini-batch=7813 updates/epoch. Mini-batch har pass mein hazaron trustworthy steps deta hai — 80/20 sweet spot.

---

C9 — Exam twist: $\eta$ ke liye linear-scaling rule

Forecast: batch $8\times$ bada ho gaya. $\eta$ ko $8$ se multiply karo, ya $\sqrt{8}$ se?

1. Rule state karo. Linear scaling: $\eta \propto B$, toh $\frac{{\eta }_{\text{new}}}{{\eta }_{\text{old}}}=\frac{B_{\text{new}}}{B_{\text{old}}}$. Yeh step kyun? Bada $B$ ⇒ chhoti gradient variance ${\sigma }^2/B$ ⇒ step zyada trustworthy hai, toh tum proportionally bolder step le sakte ho.
2. Ratio compute karo: $B_{\text{new}}/B_{\text{old}}=512/64=8$.
3. Scale karo: ${\eta }_{\text{new}}=0.01\times 8=0.08$. Yeh step kyun? Proportionality ka direct application.

Verify: $512/64=8$ ✓ aur $0.01\times 8=0.08$ ✓. (Sanity check: yeh linear rule hai, toh factor $8$, nahi $\sqrt{8}\approx 2.83$. Practice mein tum warm-up bhi add karte ho — dekho Learning Rate and Schedules — lekin exam answer $\eta =0.08$ hai.)

---

C10 — Thodi si noise saddle se kyun escape karaati hai

Forecast: $\theta =0$ par gradient hai... compute karo. Kya Batch GD kabhi move karega?

1. Batch gradient at 0: ${\nabla }_{\theta }J=3(0{)}^2=0$. Update: $\theta \leftarrow 0-0.1(0)=0$. Hamesha ke liye stuck. Yeh step kyun? Flat spot par exact gradient exactly zero hai — Batch GD ke paas move karne ki koi wajah nahi, chahe $\theta =0$ ek real minimum nahi hai (loss $\theta <0$ ke liye girta rehta hai).
2. SGD ka noisy estimate. Yaad karo (definitions se) SGD ka estimate $\hat{g}={\nabla }_{\theta }J+\epsilon$ hai, jahan $\epsilon$ sampling noise hai: mean zero, variance ${\sigma }^2$, jo kaun sa random sample draw hua usse aata hai. $\theta =0$ par mean part $0$ hai, lekin $\epsilon$ ka ek single draw generally nonzero hota hai — maano is draw se $\epsilon =+0.6$ milta hai, toh $\hat{g}=0+0.6=0.6$. Update: $\theta \leftarrow 0-0.1(0.6)=-0.06$. Yeh step kyun? Kyunki $\mathbb{E}[\epsilon ]=0$ lekin $\mathrm{Var}[\epsilon ]={\sigma }^2>0$, individual steps nonzero hote hain tab bhi jab true gradient vanish ho jaata hai. Woh nonzero kick exactly wahi hai jo $\theta$ ko flat spot se nudge karta hai — noiseless Batch GD ke liye impossible.
3. Ab true gradient wapas aata hai aur humein downhill le jaata hai. $\theta =-0.06$ par: ${\nabla }_{\theta }J=3(-0.06{)}^2=0.0108>0$, toh noiseless step bhi ab continue karta hai $\theta \leftarrow -0.06-0.1(0.0108)=-0.06108$, aur negative $\theta$ ki taraf slide karta hai jahan ${\theta }^3$ (loss) decrease karta rehta hai. Yeh step kyun? Jab noise ne humein exact saddle se hata diya, gradient ab zero nahi hai, toh ordinary descent takeover karta hai aur hum real progress karte rehte hain. Noise ko sirf tie ek baar break karna tha.

Verify: $3(0{)}^2=0$ (Batch saddle par frozen) ✓; ek noisy kick $0-0.1(0.6)=-0.06$ ✓; phir $3(-0.06{)}^2=0.0108$, giving next $\theta =-0.06-0.1(0.0108)=-0.06108$, jo $-0.06$ se zyada negative hai ✓ — yaani abhi bhi downhill move kar raha hai. Conclusion: SGD ka zero-mean sampling noise parameter ko saddle se dislodge karta hai jahan Batch GD hamesha ke liye frozen baitha rehta; ek kick ke baad, deterministic descent kaam khatam karta hai. Yeh parent ka "thodi noise achhi hoti hai" concretely dikhaya gaya hai. Dekho Saddle Points and Non-Convex Optimization.

---

Recall Matrix par quick self-test

Kaun se cell(s) dikhate hain ki $\hat{g}$ ki sign akele step direction decide karti hai? ::: C1 (positive) aur C2 (negative). C7 mein, kaun sa exact $\eta$ $x_1=2$ ke liye oscillation boundary par baitha hai? ::: $\eta =2/x_1^2=0.5$. C6 mein, answer 4 updates kyun hai aur 3 kyun nahi? ::: 100 leftover samples ek partial final batch banate hain, toh hum ceiling lete hain $\lceil 1000/300\rceil =4$. C5 mein, $B=1\to 9$ jaane par noise ka std kitne factor se cut hota hai? ::: $\sqrt{9}=3$ (variance 9 se, std 3 se). $\hat{g}$ aur ${\nabla }_{\theta }{L}_i$ kab coincide karte hain, aur kab differ karte hain? ::: Woh SGD ke liye coincide karte hain ($B=1$, ek term ka average woh term hi hota hai); mini-batch aur Batch ke liye differ karte hain, jahan $\hat{g}$ kaafi saare ${\nabla }_{\theta }{L}_i$ ko average karta hai. C10 mein, Batch GD $\theta =0$ par kyun freeze ho jaata hai lekin SGD nahi? ::: Wahan exact gradient 0 hai, toh Batch kabhi move nahi karta; SGD ka zero-mean noise $\epsilon$ ek nonzero kick deta hai jo flat spot se escape karaata hai.