Batch, mini-batch, stochastic gradient descent: Exercises

5.6.10 · D4 · Coding › Machine Learning (Aerospace Applications) › Batch, mini-batch, stochastic gradient descent

Parent note se ek update rule pe sab kuch tika hai:

$θ \leftarrow θ - η \overset{g}{^}, \overset{g}{^} = our estimate of the true gradient \nabla_{θ} J .$

Yahan $θ$ (theta) wo parameter hai jo hum tune karte hain, $η$ (eta) learning rate hai (step kitna bada ho), aur $\overset{g}{^}$ ("g-hat") estimated downhill direction hai. Neeche har symbol inhi mein se ek hai. Agar koi unfamiliar lage to Gradient Descent aur Learning Rate and Schedules dekho.

Definition Problem 1 se pehle char aur zaroori words

==Batch size $B$ == — un training examples ki sankhya jinke gradients ko average karke ek estimate $\overset{g}{^}$ banayi jaati hai, ek hi step lene se pehle. Bada $B$ = zyada examples per step.
Epoch — saare $N$ training samples se ek poora pass. Agar aap $B$ ke chunks mein update karte hain, to ek epoch mein $N / B$ updates hoti hain.
==Per-sample gradient variance $σ^{2}$ == — ek number jo batata hai ki individual examples downhill direction ke baare mein kitna alag-alag sochte hain. Agar har sample identically point karta, to $σ^{2} = 0$ ; jitna zyada scatter, utna bada $σ^{2}$ . Yeh ek single gradient ka "noise level" hai.
==Estimate variance $Var [\overset{g}{^}]$ == — hamari averaged estimate kitni noisy hai. $B$ scattered terms ko average karna ise shrink karta hai: $Var [\overset{g}{^}] = σ^{2} / B$ .

Level 1 — Recognition

L1.1 · Flavour pehchano

$N = 60, 000$ images ke dataset ke liye, aap har update se pehle saare 60,000 examples use karke gradient compute karte ho. Batch / SGD / Mini-batch mein se kaun sa hai yeh?

Recall Solution

Batch size $B$ , $N$ ke barabar hai ( $B = 60, 000 = N$ ). Parent definition ke anusaar == $B = N$ matlab Batch GD== hai. Yeh har step pe exact average gradient use karta hai — zero noise, maximum cost per step.

L1.2 · Batch size padhna

Ek engineer likhta hai: "Main 64 random samples leta hoon, unke gradients average karta hoon, phir step leta hoon." $B$ kya hai, aur yeh kaun sa flavour hai?

Recall Solution

$B = 64$ . Kyunki $1 < 64 < N$ (yeh maante hue ki dataset 64 se bada hai), yeh Mini-batch GD hai. 64 gradient terms ka average ek unbiased estimate hai jiska noise (variance $σ^{2} /64$ , jahan $σ^{2}$ upar define ki gayi per-sample gradient variance hai) ek single sample se 64 guna kam hai.

L1.3 · Noise match karo

Batch, Mini-batch ( $B = 32$ ), aur SGD ko gradient estimate ke least noisy se most noisy order mein rank karo.

Recall Solution

Estimate variance $Var [\overset{g}{^}] = σ^{2} / B$ hai (yaad karo $σ^{2}$ per-sample gradient variance hai), isliye bada $B$ = kam noise:

Batch ( $B = N$ ): least noisy (exact, variance $\to 0$ ).
Mini-batch ( $B = 32$ ): medium.
SGD ( $B = 1$ ): most noisy. Least → most order: Batch < Mini-batch < SGD.

Level 2 — Application

L2.1 · Ek SGD step

$y = θ x$ fit karo squared loss $L_{i} = \frac{1}{2} (θ x_{i} - y_{i})^{2}$ se, jahan $x_{i}$ input hai aur $y_{i}$ sample $i$ ka target hai. Data point $(x_{1}, y_{1}) = (3, 6)$ , start $θ = 1$ , learning rate $η = 0.05$ . Ek SGD step ke baad naya $θ$ compute karo.

Recall Solution

Gradient (chain rule): $\nabla_{θ} L_{1} = (θ x_{1} - y_{1}) x_{1} = (1 \cdot 3 - 6) \cdot 3 = (- 3) (3) = - 9$ . Update: $θ \leftarrow 1 - 0.05 (- 9) = 1 + 0.45 = 1.45$ . Kyun upar? Gradient negative hai, isliye iske against step lena (negative subtract karna) $θ$ ko true value $θ^{true} = 2$ ki taraf badhata hai.

L2.2 · Ek Batch step

Wohi model. Ab dono points use karo: $(x_{2}, y_{2}) = (1, 3)$ add karo, $θ = 1$ aur $η = 0.05$ rakhho. Batch update compute karo.

Recall Solution

$\nabla L_{1} = - 9$ (L2.1 se). $\nabla L_{2} = (θ x_{2} - y_{2}) x_{2} = (1 \cdot 1 - 3) \cdot 1 = - 2$ . Batch gradient ( $N = 2$ ka average): $\overset{g}{^} = \frac{1}{2} (- 9 + (- 2)) = \frac{1}{2} (- 11) = - 5.5$ . Update: $θ \leftarrow 1 - 0.05 (- 5.5) = 1 + 0.275 = 1.275$ . Average kyun, sum nahi? Batch GD mean loss gradient $\frac{1}{N} \sum \nabla L_{i}$ estimate karta hai, isliye do alag-alag samples ( $- 9$ aur $- 2$ ) blend hokar ek trustworthy direction $- 5.5$ bante hain. SGD se step chhota kyun? Mild sample ( $- 2$ ) average ko zero ki taraf khichta hai, isliye batch ek gentle, less jumpy step ( $+ 0.275$ ) leta hai — jo akele single hot sample se zyada hota ( $+ 0.45$ ) — exactly wo smoothing jo kam noise deti hai.

L2.3 · Epoch bookkeeping

Tumhare paas $N = 50, 000$ samples hain aur tum $B = 250$ choose karte ho. Ek epoch (saare data ka ek poora pass) mein kitne parameter updates hote hain? 10 epochs mein kitne?

Recall Solution

Updates per epoch $= N / B = 50, 000/250 = 200$ . 10 epochs mein $= 200 \times 10 = 2000$ updates. Kyun? Ek epoch ka matlab hai har sample ko ek baar dekhna; har mini-batch $B$ samples consume karta hai, isliye data cover karne ke liye $N / B$ batches chahiye.

L2.4 · Batch size se variance

Per-sample gradient variance $σ^{2} = 8$ hai. SGD ( $B = 1$ ), mini-batch $B = 4$ , aur mini-batch $B = 16$ ke liye $Var [\overset{g}{^}]$ compute karo.

Recall Solution

$Var [\overset{g}{^}] = σ^{2} / B$ use karo.

$B = 1$ : $8/1 = 8$ .
$B = 4$ : $8/4 = 2$ .
$B = 16$ : $8/16 = 0.5$ . $B$ ko 4 se 16 tak quadruple karne se variance 4× cut hua — saaf $1/ B$ shrinkage.

Figure — Batch, mini-batch, stochastic gradient descent

L2.5 · Jab $N$ , $B$ se divisible nahi ho

Tumhare paas $N = 1030$ samples hain aur tum $B = 250$ pick karte ho. Ek epoch mein kitne mini-batches hote hain, aur har ek kitna bada hai?

Recall Solution

$1030/250 = 4.12$ , jo poora number nahi hai. Tum $⌊ 1030/250 ⌋ = 4$ full batches of 250 banate ho (1000 samples use karke), baaki remainder $1030 - 1000 = 30$ samples reh jaate hain. Standard practice: size 30 ka ek extra chhota last mini-batch banao. To epoch mein $5$ updates hoti hain: chaar $B = 250$ ke saath aur ek $B = 30$ ke saath. Yeh kyun matter karta hai? Us last batch ki variance zyada hai ( $σ^{2} /30$ vs $σ^{2} /250$ ), isliye uska step noisier hai — kuch libraries remainder ko drop karne deti hain (drop_last=True) taaki har step equally clean rahe. Dono tarike se, epoch per updates ki sankhya $⌈ N / B ⌉ = ⌈ 4.12 ⌉ = 5$ hai.

Level 3 — Analysis

L3.1 · Wall-clock, step count nahi

Maan lo ek sample ka gradient compute karne mein $1$ unit time lagta hai. Dataset $N = 1000$ . Per epoch compare karo: (a) time cost, (b) number of updates, Batch vs. mini-batch $B = 50$ ke liye. Kaun zyada progress per unit time karta hai, aur kyun?

Recall Solution

Per epoch dono methods saare 1000 samples ek baar process karte hain, isliye dono ka cost $\approx 1000$ time units per epoch hai (same total gradient computations).

Batch: $N / N = 1$ update per epoch.
Mini-batch $B = 50$ : $1000/50 = 20$ updates per epoch. Same compute, lekin mini-batch 20 course-corrections karta hai jabki Batch sirf 1. Har mini-batch step "good enough" hai (unbiased, moderate noise), isliye 20 good-enough steps typically loss ko ek perfect step se kahin zyada neeche lete jaate hain. Isliye mini-batch wall-clock time mein jeetta hai, bhaale uski per-step direction kam exact ho.

L3.2 · Jab noise help karta hai

Non-convex loss pe do runs (dekho Saddle Points and Non-Convex Optimization): Run A Batch GD use karta hai, Run B SGD. Run A flat loss curve ke saath ruk jaata hai; Run B ka loss wiggle karta hai lekin girta rehta hai. Explain karo ki Run A ke saath kya hua hoga aur Run B kyun escape kar paaya.

Recall Solution

Run A (Batch) shayad ek saddle point ya shallow local minimum pe atak gaya: wahan exact gradient $\nabla J \approx 0$ hota hai, isliye update $θ \leftarrow θ - η \cdot 0$ barely move karta hai — yeh stall ho jaata hai. Run B (SGD) ka noisy estimate $\overset{g}{^} = \nabla L_{k}$ hai jiska variance nonzero hota hai even jab $\nabla J = 0$ ho. Yeh random kick $θ$ ko flat saddle se off push karta hai kisi slope pe jahan gradient phir se informative hai. To noise, precision ke liye harmful, flat traps escape karne ke liye helpful hai. Yahi central reason hai ki noisy small-batch methods aerospace ML ke non-convex surrogate models ke liye prefer ki jaati hain.

L3.3 · $B$ pe diminishing returns

Per-sample variance $σ^{2} = 100$ hai. Standard deviation $Var [\overset{g}{^}]$ mein reduction compute karo $B : 1 \to 4$ jaane pe, aur alag se $B : 64 \to 256$ jaane pe. 80/20 point pe comment karo.

Recall Solution

Standard deviation $= σ^{2} / B = 10/ B$ .

$B : 1 \to 4$ : $10/1 = 10 \to 10/2 = 5$ . Drop of $5$ (half ho gaya).
$B : 64 \to 256$ : $10/8 = 1.25 \to 10/16 = 0.625$ . Drop of $0.625$ . Same 4× increase in $B$ , lekin pehli jump doosri se 8× zyada noise remove karti hai. Kyunki noise $1/ B$ ki tarah girta hai, $B$ ki early increases hugely pay off karti hain aur baad ki barely kuch kharti hain — yahi 80/20 sweet spot hai jo $B \in [32, 256]$ ko justify karta hai.

Level 4 — Synthesis

L4.1 · Training budget design karo

Tumhare paas $N = 20, 000$ CFD drag samples hain, exactly 1000 gradient computations total ka compute budget hai, aur tum variance $\leq σ^{2} /10$ rakhte hue zyada se zyada parameter updates chahte ho. Per-sample variance $σ^{2}$ hai. $B$ choose karo aur un updates ki sankhya compute karo jo tum afford kar sakte ho.

Recall Solution

Constraint 1 (variance): $σ^{2} / B \leq σ^{2} /10 \Rightarrow B \geq 10$ chahiye. To smallest allowed batch $B = 10$ hai. Constraint 2 (budget): har update $B$ gradient computations cost karta hai; total budget 1000 hai. Updates $= \frac{1000}{B}$ . Updates maximise karne ke liye hum smallest legal $B$ chahte hain, yaani $B = 10$ . Updates $= 1000/10 = 100$ updates. Smallest legal B kyun? Kam computations per step matlab same budget mein zyada steps, aur $B = 10$ noise cap ko exactly satisfy karta hai. Koi bhi $B < 10$ variance requirement todega; koi bhi $B > 10$ unnecessary noise reduction pe budget waste karega.

L4.2 · Linear scaling rule in action

Ek model $B = 32$ aur $η = 0.01$ ke saath stably train karta hai. Tum bigger GPU use karne ke liye $B = 256$ switch karte ho. Linear scaling rule $η \propto B$ use karke, kya learning rate use karni chahiye, aur kyun ab bada $η$ safe hai?

Recall Solution

Ratio $256/32 = 8$ hai, isliye learning rate 8 se scale karo: $η_{new} = 0.01 \times 8 = 0.08$ . Kyun safe? Bada $B$ gradient variance shrink karta hai ( $σ^{2} /256$ vs $σ^{2} /32$ — 8× kam). Clean, more trustworthy direction $\overset{g}{^}$ ke saath, tum noise ki wajah se raaste se bhatke bina bolder step le sakte ho. $B = 256$ ke saath $η = 0.01$ rakhne se low-noise estimate tiny, over-cautious steps pe waste hogi. Zyada saaf gradient ⇒ bolder step. Adaptive alternatives ke liye Momentum and Adam dekho.

L4.3 · Multi-step mini-batch trace

Model $y = θ x$ , loss $L_{i} = \frac{1}{2} (θ x_{i} - y_{i})^{2}$ (input $x_{i}$ , target $y_{i}$ ), $η = 0.1$ , start $θ = 0$ . Do mini-batches: Batch A $= {(2, 4)}$ , Batch B $= {(1, 1), (2, 4)}$ . Pehle A pe ek update karo, phir B pe (isi order mein). Har ke baad $θ$ do.

Recall Solution

Update 1 (Batch A, $B = 1$ ): $\nabla L = (θ \cdot 2 - 4) \cdot 2 = (0 - 4) (2) = - 8$ . 1 sample pe mean $= - 8$ . $θ \leftarrow 0 - 0.1 (- 8) = 0.8$ . Update 2 (Batch B, $B = 2$ ), ab $θ = 0.8$ : $\nabla L_{(1, 1)} = (0.8 \cdot 1 - 1) \cdot 1 = - 0.2$ . $\nabla L_{(2, 4)} = (0.8 \cdot 2 - 4) \cdot 2 = (1.6 - 4) (2) = - 4.8$ . Mean $\overset{g}{^} = \frac{1}{2} (- 0.2 + (- 4.8)) = - 2.5$ . $θ \leftarrow 0.8 - 0.1 (- 2.5) = 0.8 + 0.25 = 1.05$ . Parameter $0 \to 0.8 \to 1.05$ climb karta hai $θ^{true} = 2$ ki taraf, doosra (averaged) step pehle se smoother hai.

Level 5 — Mastery

L5.1 · Convergence neighbourhood & schedules

Rigorously explain karo ki constant- $η$ SGD sirf minimum ke ek neighbourhood tak kyun converge karta hai, exact point tak nahi, aur ek learning-rate schedule ise kaise fix karta hai. Update rule se illustrate karo.

Recall Solution

True minimum $θ^{opt}$ pe, full gradient $\nabla J (θ^{opt}) = 0$ hota hai, lekin ek single-sample estimate $\overset{g}{^} = \nabla L_{k}$ ki wahan bhi nonzero variance $σ^{2}$ hoti hai. To update $θ \leftarrow θ^{opt} - η \overset{g}{^}$ phir bhi typical size $η σ$ se random move karta hai. Iterate kabhi settle nahi karta; yeh $θ^{opt}$ ke around $\sim η σ$ radius ki ball ke andar bounce karta rehta hai. Bada $η$ ⇒ badi ball. Fix — ek schedule $η_{t} \to 0$ (e.g. $η_{t} = η_{0} / t$ ): jaise steps badhte hain, bounce amplitude $η_{t} σ \to 0$ hoti hai, isliye neighbourhood shrink hokar ek point ban jaata hai. Classic Robbins–Monro conditions $\sum_{t} η_{t} = \infty$ (kaafi door travel karo) aur $\sum_{t} η_{t}^{2} < \infty$ (noise damp out ho) $θ^{opt}$ tak convergence guarantee karte hain. Learning Rate and Schedules dekho.

L5.2 · Sharp vs. flat minima trade-off

Large batches often small ones se zyada bura generalise karte hain, training loss kam hone ke bावजूद. Reason through karo kyun, batch noise, minima geometry, aur Bias-Variance Tradeoff ko connect karke.

Recall Solution

Ek loss surface mein sharp minima (narrow, steep walls) aur flat minima (wide basins) hote hain. Sharp minima training set ko tightly fit karte hain lekin train aur test data ke beech ek chhoti si shift tumhe ek steep wall pe le jaati hai → bada test loss (poor generalisation). Flat minima robust hote hain: nearby points ka loss bhi kam hai, isliye train/test shift barely hurt karta hai. Batch size control karta hai ki tum kaun sa paate ho. Large-batch gradients almost noise-free hote hain, isliye optimiser seedha nearest minimum mein slide kar jaata hai — aksar ek sharp wala. Small-batch noise (variance $σ^{2} / B$ ) ek random kick ki tarah act karta hai jo ek narrow sharp basin ke andar nahi reh sakta — yeh shake out ho jaata hai aur sirf wide flat basins ise hold kar sakti hain. Isliye small batches implicitly flat, generalising minima ki taraf bias karte hain. Bias–variance terms mein: noise ek form of regularisation hai jo thodi si training-set fit (slightly higher bias) trade karta hai kahin better test performance ke liye (learned function ki lower variance). Isliye practice mein mid-size batches dominate karte hain, huge wale nahi.

L5.3 · Aerospace pipeline ke liye method choose karo

Ek surrogate neural net wing geometry se drag predict karta hai. Training data continuously live wind-tunnel sensors se stream in ho raha hai (tum sab store nahi kar sakte). Kaun sa method — Batch, Mini-batch, ya SGD — aur kyun? Memory, noise, aur streaming nature address karo.

Recall Solution

Batch GD impossible hai: ise $\frac{1}{N} \sum \nabla L_{i}$ form karne ke liye memory mein saare $N$ samples chahiye, lekin data stream in hota rehta hai aur fully store nahi ho sakta — $N$ effectively unbounded/growing hai. Pure SGD ( $B = 1$ ) streaming ke saath kaam karta hai (per sample update) lekin bahut noisy hai aur har GPU pass ko under-use karta hai. Mini-batch sahi choice hai: incoming stream ka ek chhota window buffer karo (e.g. $B = 64$ ), update karo, discard karo, repeat karo. Yeh (a) memory mein fit hota hai, (b) ek stable-enough direction ke liye variance $σ^{2} /64$ tak reduce karta hai, (c) online/streaming setting ke liye suit karta hai, aur (d) woh useful noise rakhta hai jo non-convex surrogate landscape ke saddle points se escape karti hai. Ek decaying schedule ke saath pair karo taaki early exploration precise convergence mein badal sake, aur noisy stream ko aur smooth karne ke liye Momentum and Adam consider karo.

Parent note pe wapas jao · related: Gradient Descent, Backpropagation, Loss Functions.