Visual walkthrough — Batch, mini-batch, stochastic gradient descent

5.6.10 · D2 · Coding › Machine Learning (Aerospace Applications) › Batch, mini-batch, stochastic gradient descent

Hum parent ki central result ko re-derive kar rahe hain: gradient-descent step, aur batch size $B$ us step mein noise ko kaise control karta hai. Related ideas Gradient Descent, Learning Rate and Schedules, Saddle Points and Non-Convex Optimization, aur Loss Functions mein milti hain.

---

Step 1 — "Loss surface" kya hoti hai? (woh pahaad jis par tum khade ho)

KYA. Koi bhi formula se pehle, ek landscape imagine karo. Zameen ek curved sheet hai. Tumhara left–right position ek aisa number hai jise tum tune kar sakte ho — isse $\theta$ (pronounced "theta") kaho, hamare toy model ka ek akela knob. Sheet par tumhari height yeh hai ki model abhi kitna bura perform kar raha hai — ek number jise loss kehte hain, likha jaata hai $J(\theta )$.

• $\theta$ = horizontal position = woh parameter jo hum change kar sakte hain.
• $J(\theta )$ = us position ke bilkul upar ki height = "hum kitna galat hain."

KYUN. Har learning problem secretly yahi hai: "woh position $\theta$ dhundho jahan height $J$ sabse kam ho." Kam height = chhoti error. Toh learning = valley ke bottom tak chalna.

PICTURE. Neeche diye gaye curve ko dekho. Red dot woh jagah hai jahan hum abhi khade hain. Neeche green X goal hai — sabse lowest point, sabse achha $\theta$.

---

Step 2 — Gradient kya hota hai? (downhill kaun si taraf hai?)

KYA. Pahaad par aankhon par patti baandhe khade ho, jaanna chahte ho: kaun si taraf slope neeche hai, aur kitni steeply? Yahan tumhari jagah par jawab ek akela number hai: curve ki slope, likhi jaati hai ${\nabla }_{\theta }J$ (padhte hain "grad J"). Ek knob ke liye yeh bas $\frac{dJ}{d\theta }$ hai — us line ka tilt jo curve ko tumhare point par sirf ek jagah touch karti hai (tangent line).

• ${\nabla }_{\theta }J$ = slope = "ek tiny step right ke liye, height kitna badlegi?"
• Positive slope → curve daayein taraf upar jaati hai → downhill baayein hai.
• Negative slope → curve baayein taraf upar jaati hai → downhill daayein hai.

Slope kyun, kuch fancy kyun nahi? Kyunki slope exactly woh tool hai jo jawab deta hai "kaun si direction yahan height sabse tezi se girayegi." Ek limit-based slope (ek derivative) surface ka sabse chhota possible local description hai — yeh woh kam se kam hai jo tumhe chahiye aur woh sabse zyada hai jis par tum apne paon ke paas trust kar sakte ho.

PICTURE. Orange tangent line red dot par curve ko touch karti hai. Uski steepness hai ${\nabla }_{\theta }J$. Arrow woh direction dikhata hai jis taraf slope uphill point karta hai; downhill iska opposite hai.

---

Step 3 — Hum gradient ke opposite kyun step karte hain (Taylor argument, drawn)

KYA. Hum ek chhota step $\Delta \theta$ (baayein ya daayein ek tiny nudge) lena chahte hain jo height girayein. Hamare paon ke paas curve almost straight lagti hai (tangent line). Toh ek small nudge ke liye height mein change approximately yeh hai:

$\underset{\text{new height}}{\underbrace{J(\theta +\Delta \theta )}}\approx \underset{\text{height now}}{\underbrace{J(\theta )}}+\underset{\text{slope}}{\underbrace{{\nabla }_{\theta }J(\theta )}}\cdot \underset{\text{our nudge}}{\underbrace{\Delta \theta }}$

Term by term padho: new height ≈ old height + (slope × nudge). Woh "slope × nudge" woh akela part hai jise hum control karte hain.

Yeh equation kyun? Yeh ek Taylor expansion hai jo apne linear (straight-line) piece tak rakhi gayi hai — ek fancy naam hai "curve ko point ke paas uski tangent se replace karo." Hum ise use karte hain kyunki ek tiny step ke liye tangent aur curve mein koi fark nahi hota, toh yeh hume sachchi taur par batata hai kaun sa nudge height girayega.

Height change $\nabla J\cdot \Delta \theta$ ko ek fixed step size ke liye jitna ho sake utna negative banane ke liye, nudge ko slope ke against point karo:

$\boxed{\Delta \theta =-\eta {\nabla }_{\theta }J}$

• $\eta$ (padhte hain "eta") = ek small positive number, step size / learning rate.
• Minus sign = "downhill jao, uphill slope ke opposite."

PICTURE. Red dot se do candidate nudges: ek green wala slope ke against (height girta hai) aur ek coral wala slope ke saath (height badhta hai). Green jeet jaata hai.

---

Step 4 — Gradient actually kahan se aata hai? (yeh ek average hai)

KYA. Height $J$ ek cheez nahi hai — yeh tumhare saare training examples par average unhappiness hai. Maano tumhare paas $N$ examples (data points) hain. Har example $i$ ki apni ek chhoti loss $L_i$ hai. Tab:

$J(\theta )=\frac{1}{N}{\sum }_{i=1}^N{L}_i$

• $N$ = tumhare paas kitne training examples hain.
• ${\sum }_{i=1}^N$ = "har example par add karo, 1st se $N$th tak."
• $\frac{1}{N}$ = "count se divide karo, yaani average lo."
• $L_i$ = ek single example ki loss.

Kyunki slopes ka average, average ka slope hota hai, sach mein downhill direction bhi ek average hai:

${\nabla }_{\theta }J=\frac{1}{N}{\sum }_{i=1}^N{\nabla }_{\theta }{L}_i$

Yeh matter kyun karta hai? Kyunki woh sum hi expensive part hai. Har ${\nabla }_{\theta }{L}_i$ real compute maangta hai (ek full forward + Backpropagation pass). Agar $N$ millions mein hai, toh ek honest gradient ek chhoti eternity hai.

PICTURE. Har faint arrow ek example ka downhill direction ${\nabla }_{\theta }{L}_i$ hai. Yeh scatter karte hain. Unka average (bold arrow) true gradient ${\nabla }_{\theta }J$ hai.

---

Step 5 — Woh ek idea jo Batch / Mini-batch / SGD ko alag karta hai

KYA. Hum hamesha full average afford nahi kar sakte. Toh hum ise ek subset se estimate karte hain. Estimate ko $\hat{g}$ ("g-hat") kaho. Sabke liye general step same hai:

$\theta \leftarrow \theta -\eta \hat{g}$

Teen methods mein akela fark yeh hai ki $\hat{g}$ banane ke liye kitne examples $B$ average kiye:

• $\hat{g}=\frac{1}{N}{\sum }_{i=1}^N\nabla L_i$ — Batch: $B=N$, saare examples use karo.
• $\hat{g}=\frac{1}{B}{\sum }_{i\in \mathcal{B}}\nabla L_i$ — Mini-batch: $1<B<N$, ek chunk $\mathcal{B}$ use karo.
• $\hat{g}=\nabla L_k$ — SGD: $B=1$, ek random example $k$ use karo.

Yahan $B$ = batch size = kitne examples ek step ko feed karte hain. $\mathcal{B}$ = chosen examples ka chhota set.

Kam kyun choose karo? Kam examples → sasta $\hat{g}$ → tum same time mein bahut zyada steps le sakte ho. Par kimat yeh hai ki $\hat{g}$ ab sirf true arrow ka ek andaaza hai.

PICTURE. Per-example arrows ka wahi scatter. Batch sab ko average karta hai → true arrow par land karta hai. Mini-batch kuch ko average karta hai → kareeb-ish. SGD ek grabta hai → noticeably alag point kar sakta hai.

---

Step 6 — Andaaza unbiased kyun hota hai (average par sahi)

KYA. Ek example $k$ uniformly at random chuno (har example equally likely, chance $\frac{1}{N}$ har ek ke liye). Expected value ${\mathbb{E}}_k$ — kaafi random picks par long-run average — us single-sample gradient ka yeh hai:

${\mathbb{E}}_k[\nabla L_k]={\sum }_{k=1}^N\underset{\text{chance of }k}{\underbrace{\frac{1}{N}}}\underset{\text{its arrow}}{\underbrace{\nabla L_k}}=\frac{1}{N}{\sum }_{k=1}^N\nabla L_k={\nabla }_{\theta }J$

• ${\mathbb{E}}_k[\cdot ]$ = "agar tum random pick forever repeat karo toh average outcome."
• Result: one-sample guesses ka average true gradient ke barabar hota hai.

Ise unbiased kyun kehte hain. Unbiased matlab koi systematic tilt nahi — guesses average par baayein ya daayein nahi jhukti, yeh truth ke around scatter hoti hain. Toh SGD average par galat nahi hai; yeh bas noisy hai.

PICTURE. Ek dartboard: bullseye true gradient hai. SGD darts (ek example each) widely scatter karte hain par center bullseye par karte hain. Mini-batch darts uske paas tightly cluster karte hain. Woh clustering-vs-scatter hi agley step ki poori kahani hai.

---

Step 7 — Zyada examples matlab kam noise ($1/B$), drawn

KYA. Maano har single example ka gradient truth ke around ek spread ke saath wobble karta hai jise hum variance ${\sigma }^2$ ("sigma-squared" = scatter measure karne wala ek number) kehte hain. Agar hum $B$ independent examples average karein, us average ka scatter shrink ho jaata hai:

$\mathrm{Var}[\hat{g}]=\frac{{\sigma }^2}{B}$

• $\mathrm{Var}[\hat{g}]$ = hamare estimate ka step to step kitna jiggle hota hai.
• ${\sigma }^2$ = ek single example ka jiggle.
• $B$ se divide karna = "jitna zyada average karo, result utna calm."

$1/B$ kyun, $1/B^2$ kyun nahi? Averaging independent wobbles ko partially cancel karta hai, perfectly nahi — errors partly overlap karte hain. $B$ independent numbers ko average karne ka math exactly variance mein $\frac{1}{B}$ shrink deta hai (actual wiggle length mein $\frac{1}{\sqrt{B}}$ shrink). Woh ​ hi reason hai ki $B$ ko $256$ se $512$ tak double karna barely help karta hai — dekho Bias-Variance Tradeoff.

PICTURE. Ek hi valley par teen descent paths: SGD ($B=1$) wildly zig-zag karta hai, mini-batch ($B=32$) gently wiggle karta hai, Batch ($B=N$) smooth glide karta hai par kam steps leta hai. Same destination, alag alag personalities.

---

Step 8 — Degenerate aur edge cases (koi gap mat chhodna)

Har scenario, drawn taaki tum kabhi surprise se na milo:

• $B=N$ (Batch, maximum). Subset hi poora dataset hai, toh $\hat{g}=\nabla J$ exactly. Variance ${\sigma }^2/N$ — jitna ho sake utna chhota. Ek update per epoch.
• $B=1$ (SGD, minimum). Variance ${\sigma }^2/1={\sigma }^2$ — maximum jitter. $N$ updates per epoch.
• $B>N$? Impossible — tum apne paas se zyada examples average nahi kar sakte. $B$ $N$ par capped hai.
• $\eta$ bahut bada. Step 3 ki linear (tangent) approximation sirf small nudges ke liye hold hoti hai. Ek huge $\eta$ valley ko overshoot karta hai aur loss climb karti hai — divergence. Dekho Learning Rate and Schedules.
• $\eta \to 0$. Steps infinitesimally tiny ho jaate hain; tum crawl karte ho aur effectively kabhi nahi pahunchte. Kuch step size zaroori hai.
• Zero gradient ($\nabla J=0$). Tum flat spot par ho: ek minimum (done!), ek maximum, ya ek saddle. Batch GD yahan freeze ho jaata hai; SGD ka noise abhi bhi saddle se nudge kar sakta hai.
• Bumpy valley (non-convex). Kai dips exist karte hain; mila hua path noise aur start point par depend karta hai. Chhota $B$ zyada explore karta hai.

PICTURE. 2×2 gallery: (a) $\eta$ achha → clean descent, (b) $\eta$ bahut bada → overshoot/diverge, (c) flat saddle jahan Batch freeze ho jaata hai par SGD escape kar jaata hai, (d) two-valley landscape jahan noise deeper wala chunti hai.

---

Ek-picture summary

Upar sab compressed: loss curve, tangent slope, anti-slope step, aur teen batch-size personalities same valley par alag alag jitter ke saath utarti hain — smooth Batch, gentle mini-batch, wild SGD — sab $\theta \leftarrow \theta -\eta \hat{g}$ aur $\mathrm{Var}[\hat{g}]={\sigma }^2/B$ follow karte hue.

Recall Feynman: poora walkthrough seedhe shabdon mein

Tum aankhon par patti baandhe ek curved pahaad par ho aur sabse neechi jagah chahte ho. (1–2) Tum apne paon ke neeche zameen ka tilt mehsoos karte ho — woh tilt hi gradient hai. (3) Neeche jaane ke liye, tum tilt ke against step karte ho; ek tiny step ki height change bas tilt × step hai, jo sabse negative hoti hai jab tum seedha opposite jaate ho. (4) Par "true tilt" actually tumhare saare training examples par average tilt hai — aur millions par average karna exhausting hai. (5) Toh tum cheat karte ho: bas kuch jagah (ya ek) par tilt mehsoos karo aur us guess par step karo. Batch saari jagah mehsoos karta hai (perfect, slow), SGD ek mehsoos karta hai (fast, wild), mini-batch ek mutthi bhar mehsoos karta hai (smart middle). (6) Ek-jagah guess biased nahi hai — kaafi taps par yeh truth par average karta hai — yeh bas noisy hai. (7) $B$ taps average karna noise ko $1/B$ se calm karta hai, toh bade batches seedha chalte hain par kam baar step karte hain. (8) Thodi dash of noise helpful bhi hoti hai: yeh tumhe flat saddles se bahar kick karti hai jahan ek perfect walker stuck ho jaata. Yahi poori idea hai.

Recall

Ek sentence mein, Batch, mini-batch, aur SGD mein akela fark kya hai? ::: Kitne examples $B$ ko har step se pehle gradient $\hat{g}$ estimate karne ke liye average kiya jaata hai. Hum $-\nabla J$ direction mein kyun step karte hain? ::: Taylor kehta hai $\Delta J\approx \nabla J\cdot \Delta \theta$, jo sabse zyada negative hota hai jab step slope ke anti-parallel ho. $\hat{g}$ ki noise batch size par kaise depend karti hai? ::: $\mathrm{Var}[\hat{g}]={\sigma }^2/B$ — zyada examples, kam jitter. Single-sample gradient "unbiased" kyun hai? ::: ${\mathbb{E}}_k[\nabla L_k]=\nabla J$; average par ek random example sahi direction point karta hai.