Noesis
5.6.10 · D5 · HinglishMachine Learning (Aerospace Applications)

Question bank — Batch, mini-batch, stochastic gradient descent

1,579 words7 min read↑ Read in English

5.6.10 · D5 · Coding › Machine Learning (Aerospace Applications) › Batch, mini-batch, stochastic gradient descent

Dive karne se pehle, teen words jo tumhare paas already hone chahiye (sab parent note mein build hue hain):

  • Gradient estimate — true downhill direction ka humara andaaza, data ke ek subset se banaya gaya.
  • Batch size — kitne training examples ek mein jaate hain (aur isliye ek parameter step mein).
  • ==Variance of == — kitna step-to-step true gradient ke around jitter karta hai; yeh ki tarah scale karta hai.

Yahan sab kuch inn par aur Gradient Descent, Learning Rate and Schedules, aur Saddle Points and Non-Convex Optimization ke non-convex landscapes par depend karta hai.

True or false — justify

"Batch GD hamesha SGD se lower final loss tak pahunchta hai."
False. Batch GD har step mein exact gradient find karta hai lekin kam steps leta hai; SGD ka noise aksar flatter, better-generalising minima mein land karta hai, toh final loss comparable ya better hoti hai — dekho Bias-Variance Tradeoff.
"SGD exactly ek example per epoch use karta hai."
False. SGD ek example per update use karta hai; ek epoch saare examples ka ek full pass hota hai, toh pure SGD updates per epoch karta hai.
"Batch size ko 1 se 32 aur 256 se 512 tak badhana noise ko same factor se reduce karta hai."
False. Variance ki tarah girta hai, toh isko 32 guna cut karta hai jabki sirf half karta hai — diminishing returns hi woh reason hai ki mid-size batches jeette hain.
"Mini-batch GD ek compromise hai, toh yeh dono extremes se slower converge karta hoga."
False. Yeh wall-clock time mein aksar dono ko beat karta hai: Batch se saste steps, SGD se kam wasted jitter, aur GPU-friendly parallel arithmetic.
"Update rule teeno methods ke liye alag hai."
False. Rule identical hai; sirf ki definition (jis subset par yeh average karta hai) badlti hai.
"Ek bada batch hamesha tumhe same learning rate safely rakhne deta hai."
False. Bada gradient variance ko shrink karta hai, toh tum bada afford kar sakte ho; linear scaling rule kehta hai . ko fixed rakhna cleaner gradient ko under-use karna hai. Dekho Learning Rate and Schedules.
"Kyunki SGD unbiased hai, uska har single step downhill point karta hai."
False. Unbiased ka matlab hai ki average par yeh downhill point karta hai (); koi bhi individual step sideways ya uphill bhi point kar sakta hai.
"Fixed learning rate ke saath, SGD exact minimum tak converge karta hai."
False. Fixed SGD ko minimum ke neighbourhood mein hamesha ke liye bouncing chodta hai; sirf decaying schedule true point ko pin karta hai.

Spot the error

Claim: "Batch gradient ."
average missing hai. Iske bina dataset size ke saath grow karta hai, effective step ko wildly -dependent banata hai — definition mean hai.
Claim: "For SGD, for one random ."
Koi factor wahan belong nahi karta. SGD ka estimate sirf hai; se divide karna har step ko se shrink kar deta aur unbiasedness tod deta.
Claim: "The step is because we descend."
Sign error. Descend karne ka matlab hai gradient ke opposite jaana: . Plus sign loss ko climb karta hai.
Claim: ", gives updates per epoch."
Ratio ulta hai. Updates per epoch hote hain, ya nahi.
Claim: "Variance of scales like , so noise drops slowly."
Variance khud ki tarah scale karta hai; standard deviation (square root) ki tarah scale karti hai. Dono ko mix karna trade-off ko galat state karta hai.
Claim: "The Taylor step used the second-order term to justify descent."
Descent direction argument ko sirf first-order term chahiye; Hessian term curvature methods ka hissa hai, plain gradient-descent derivation ka nahi.
Claim: "Averaging i.i.d. gradients halves the variance when doubles, so it is linear in ."
Yeh sach mein hai, jo doubling par variance half karta hai — lekin "linear in " phrase galat wording hai: variance ke inversely proportional hai, linearly proportional nahi.

Why questions

Hum hamesha exact full-data gradient compute karne ki jagah ko subset se approximate kyun karte hain?
Jab bahut bada hota hai (millions of CFD ya sensor samples), tab saare samples ka exact sum bahut expensive hota hai; ek subset per step bahut saste tarike se ek good-enough direction deta hai.
Thoda gradient noise kabhi kabhi harmful ki jagah helpful kyun hota hai?
Noise parameters ko saddle points aur shallow local minima se bahar bounce kar sakta hai jo non-convex nets mein problem banate hain, training ko better regions tak pahunchne deta hai — yeh Saddle Points and Non-Convex Optimization se related hai.
Step specifically gradient ke anti-parallel kyun point karta hai?
First-order Taylor term se, dot product tab sabse negative hota hai (fastest decrease) jab ka oppose kare, Cauchy–Schwarz se.
Bahut bade batches cleaner gradients ke bawajood worse generalise kyun kar sakte hain?
Kam noise sharp minima mein settle hota hai jo training data ko tightly fit karta hai lekin poorly transfer karta hai; kuch noise flatter, more robust minima ki taraf nudge karta hai — yeh Bias-Variance Tradeoff ek disguise mein hai.
Mini-batch aksar Batch GD ko wall-clock time mein kyun beat karta hai jabki har Batch step "more correct" hota hai?
Batch sirf ek costly update per epoch karta hai; mini-batch usi pass mein cheap, good-enough updates karta hai, toh per second total progress zyada hoti hai.
SGD ke liye truly converge karne ke liye learning rate eventually kyun shrink honi chahiye?
Ek fixed stochastic bounce ko minimum ke around alive rakhta hai; ek schedule us bounce ko damp karta hai toh iterates actual minimum par settle ho jaate hain. Dekho Learning Rate and Schedules.

Edge cases

Jab dataset mein exactly ek sample ho, , toh SGD kya hai?
Teeno methods same cheez mein collapse ho jaate hain: Batch, SGD, aur mini-batch teeno us single example ko use karte hain, toh identical aur exact hota hai har step mein.
Jab ho toh ka kya hota hai?
Yeh full-data average ka variance ban jaata hai, is data se achieve hone wala sabse chota; true objective ke liye estimate exact hota hai, toh sampling noise vanish ho jaati hai.
Agar se divisible nahi hai (maano , ), toh per epoch kitne updates?
Tumhe updates milte hain: 300 ke teen full batches plus 100 ka ek leftover batch (final batch simply smaller hota hai).
Agar ko se bada set karo toh effective batch behaviour kya hai?
Tum se zyada distinct samples draw nahi kar sakte, toh yeh Batch GD () par cap ho jaata hai; maangna sirf data reuse karta hai ya full set tak clamp ho jaata hai.
Saddle point par jahan exactly ho, wahan har method kya karta hai?
Batch GD ka hota hai toh yeh indefinitely stall karta hai; SGD/mini-batch mein nonzero noise hai ( per sample) jo isko saddle se kick karta hai aur training continue hone deta hai.
batch size se independent kyun kuch nahi karta?
Update bilkul koi change nahi karta — parameters freeze ho jaate hain, toh koi learning nahi hoti chahe kitna bhi accha ho.
Jab (infinite i.i.d. data per step), toh mini-batch gradient kya approach karta hai?
Uski variance , toh true expected gradient tak converge karta hai — mini-batch direction mein exact Batch GD se indistinguishable ho jaata hai.
Recall Tab band karne se pehle ek-line self-test

Answers cover karo aur re-derive karo: (1) updates per epoch ; (2) ; (3) step direction ; (4) unbiased ka matlab hai, "hamesha correct" nahi. Agar charon instantly aaye, toh topic tumhara hai.