Batch, mini-batch, stochastic gradient descent

5.6.10 · Coding › Machine Learning (Aerospace Applications)

YEH farq exist hi kyun karta hai?

Jo loss hum actually minimise karna chahte hain woh saare $N$ training samples ka average hai:

$J (θ) = \frac{1}{N} \sum_{i = 1}^{N} L (f_{θ} (x_{i}), y_{i})$

Neeche jaane ke liye hume gradient $\nabla_{θ} J$ chahiye. Exact gradient yeh hai:

$\nabla_{θ} J (θ) = \frac{1}{N} \sum_{i = 1}^{N} \nabla_{θ} L_{i}$

Problem yeh hai: jab $N$ bahut bada ho (millions of aerodynamic CFD samples, sensor logs, etc.) toh yeh pura sum compute karna expensive hai. Isliye hum gradient ko data ke ek subset se approximate karte hain. Teen methods mein subset size ke teen choices hain.

Update rule kaise derive hota hai (first principles se)

KIYA chahiye: parameters $θ$ jo $J (θ)$ ko minimise karein.

Step 1 — Loss ko Taylor expand karo current $θ$ ke aaspaas ek chhote step $Δ θ$ ke liye:

$J (θ + Δ θ) \approx J (θ) + \nabla_{θ} J (θ)^{⊤} Δ θ$

Yeh step kyun? $θ$ ke paas, loss locally linear hoti hai; linear term change ko dominate karta hai.

Step 2 — $Δ θ$ choose karo taaki $J$ sabse tezi se ghate. $\nabla J^{⊤} Δ θ$ ko ek fixed step length ke liye jitna ho sake utna negative banane ke liye, gradient ke opposite direction mein chalo:

$Δ θ = - η \nabla_{θ} J (θ)$

Yeh step kyun? Dot product $\nabla J^{⊤} Δ θ$ minimize hota hai jab $Δ θ$ anti-parallel ho $\nabla J$ ke (Cauchy–Schwarz). $η > 0$ learning rate hai (step size).

Step 3 — General update:

$θ \leftarrow θ - η \overset{g}{^}$

jahan $\overset{g}{^}$ hamara $\nabla_{θ} J$ ka estimate hai. Teeno methods mein sirf $\overset{g}{^}$ alag hai:

SGD "unbiased" kyun hai? Agar hum $k$ uniformly at random ${1, \dots, N}$ se choose karein:

$E_{k} [\nabla_{θ} L_{k}] = \sum_{k = 1}^{N} \frac{1}{N} \nabla_{θ} L_{k} = \nabla_{θ} J$

Toh average par ek random sample sahi direction mein point karta hai — bas noisy hota hai. Wahi noise sab kuch hai.

Variance trade-off (80/20 core)

Size $B$ ke mini-batch ke liye jo data se drawn hai jismein per-sample gradient variance $σ^{2}$ hai:

$Var [\overset{g}{^}] = \frac{σ ^{2}}{B}$

Kyun? $B$ i.i.d. terms ke average ka variance $1/ B$ ki tarah shrink karta hai.

Figure — Batch, mini-batch, stochastic gradient descent

Worked Example 1 — haath se ek SGD step

$y = θ x$ fit karo squared loss $L_{i} = \frac{1}{2} (θ x_{i} - y_{i})^{2}$ ke saath. Data: $(x_{1}, y_{1}) = (2, 4)$ , start $θ = 0$ , $η = 0.1$ .

Step: $\nabla_{θ} L_{1} = (θ x_{1} - y_{1}) x_{1} = (0 \cdot 2 - 4) \cdot 2 = - 8$ . Yeh step kyun? Chain rule: $\frac{d}{d θ} \frac{1}{2} (θ x - y)^{2} = (θ x - y) \cdot x$ .

Update: $θ \leftarrow 0 - 0.1 (- 8) = 0.8$ . Kyun? Gradient negative hai, isliye hum $θ$ mein upar step karte hain true value $θ^{*} = 2$ ki taraf.

Humne ek sample use kiya — yeh SGD hai.

Worked Example 2 — ek Batch step

Same model, ab dono points use karo: $(x_{2}, y_{2}) = (1, 1)$ add karo, $θ = 0$ .

$\nabla L_{1} = - 8$ (jaise upar). $\nabla L_{2} = (0 \cdot 1 - 1) \cdot 1 = - 1$ .

Batch gradient: $\overset{g}{^} = \frac{1}{2} (- 8 + - 1) = - 4.5$ . Average kyun? Batch GD saare $N = 2$ samples par mean use karta hai — true $J$ gradient.

Update: $θ \leftarrow 0 - 0.1 (- 4.5) = 0.45$ . Smoother, ek trustworthy step.

Worked Example 3 — mini-batch epoch bookkeeping

$N = 1000$ samples, $B = 100$ . Ek epoch (data par ek full pass) mein kitne updates hote hain?

$updates = \frac{N}{B} = \frac{1000}{100} = 10.$ Kyun? Har mini-batch $B$ samples consume karta hai; saara data ek baar cover karne ke liye $N / B$ batches chahiye. Batch GD 1 update/epoch deta; pure SGD 1000 deta.

Common mistake Steel-manned misconceptions

"Batch GD best hai kyunki exact gradient use karta hai." Kyun sahi lagta hai: exact = optimal direction, koi noise nahi. Fix: exactness per-step hai, per-second nahi. Batch GD ek epoch mein ek bada step leta hai; mini-batch bahut saare good-enough steps leta hai aur usually wall-clock time mein tezi se converge karta hai, plus uski noise saddle points se bahar nikalti hai. Exact ≠ efficient.

"SGD ki noise matlab yeh kabhi converge nahi hoga." Kyun sahi lagta hai: loss curve hamesha bouncing dikhti hai. Fix: yeh minimum ke ek neighbourhood mein converge karta hai. Samay ke saath learning rate ghataane se ( $η_{t} \to 0$ , ek schedule) bounce khatam ho jaata hai aur true minimum pin ho jaata hai.

"Bada batch hamesha better train karta hai." Kyun sahi lagta hai: kam noise, clean curves, GPU-friendly. Fix: bahut bade batches aksar bure generalise karte hain (woh sharp minima mein land karte hain) aur noise reduction diminishing returns deta hai ( $\propto 1/ B$ ). Mid-size batches practical optimum hain.

"Learning rate kisi bhi batch size ke liye same reh sakti hai." Kyun sahi lagta hai: $η$ aur $B$ independent lagte hain. Fix: bada $B$ → chhota gradient variance → aap badi $η$ use kar sakte hain (aur karni chahiye) (linear scaling rule: $η \propto B$ ).

Recall Feynman: 12-saal ke bachche ko explain karo

Socho tum aankhon par patti baandhe ek pahaadi par ho aur neeche pohunchna chahte ho. Neeche ka rasta feel karne ke liye tum apna paon zameen par thapthaate ho.

Batch GD: tum ek kadam uthaane se pehle apne aaspaas har direction mein dhyan se slope measure karte ho. Accurate hai lekin bahut dheeeera.
SGD: tum zameen par ek baar ek random jagah thapthaate ho aur turant kadam lete ho. Kabhi kabhi thoda galat kadam uthta hai, lekin bahut saare jaldi-jaldi kadam le lete ho aur jaldi neeche pohunch jaate ho.
Mini-batch: tum mutthi bhar jagahon par thapthaate ho, unka average lete ho, phir kadam lete ho. Perfect nahi, random nahi — samajhdaari wala middle raasta. Isliye almost sab isi ka use karte hain.

Flashcards

Batch GD, SGD, aur Mini-batch GD ke liye batch size

B

kya hai?

Batch:

B = N

; SGD:

B = 1

; Mini-batch:

1 < B < N

Teeno methods ke liye ek general update rule kya hai?

θ \leftarrow θ - η \overset{g}{^}

, jahan

\overset{g}{^}

gradient estimate hai.

SGD gradient estimate ko "unbiased" kyun kehte hain?

Kyunki

E_{k} [\nabla L_{k}] = \nabla J

; average par ek random sample true gradient direction mein point karta hai.

Gradient-estimate variance batch size

B

ke saath kaise scale karta hai?

Var [\overset{g}{^}] \propto σ^{2} / B

— bada batch, kam noise.

N

samples aur batch size

B

ke saath ek epoch mein kitne parameter updates hote hain?

N / B

updates.

SGD mein thodi noise beneficial kyun ho sakti hai?

Yeh non-convex loss landscapes mein saddle points aur shallow local minima se bahar nikalti hai.

Step direction

- \nabla J

hone ka derivation reason batao.

Taylor:

Δ J \approx \nabla J^{⊤} Δ θ

; yeh sabse zyada negative hota hai jab

Δ θ

anti-parallel ho

\nabla J

ke.

Batch GD wall-clock time mein mini-batch se aksar kyun haarta hai?

Yeh ek epoch mein sirf ek (expensive) update karta hai; mini-batch bahut saare saste good-enough updates karta hai.

Learning rate ke liye linear scaling rule kya hai?

Batch size badhne par learning rate roughly proportionally scale karo:

η \propto B

Noisy SGD actually minimum par converge kaise karta hai?

Decreasing learning-rate schedule

η_{t} \to 0

se.

Connections

Gradient Descent — parent algorithm; yeh iske data-sampling variants hain.
Learning Rate and Schedules — $η$ ko batch size aur time ke hisaab se adapt karna padta hai.
Momentum and Adam — mini-batch noise ko running averages se smooth karte hain.
Loss Functions — woh $L_{i}$ jinka hum sum karte hain.
Bias-Variance Tradeoff — wohi variance logic $\overset{g}{^}$ ko govern karta hai.
Saddle Points and Non-Convex Optimization — aerospace surrogate models mein noise kyun help karti hai.
Backpropagation — har $\nabla_{θ} L_{i}$ actually kaise compute hota hai.