3.2.2
Intuition Ek hi saanch mein core ideaNetwork train karne ke liye humein poore dataset ka loss gradient chahiye. Lekin usse exactly compute karna expensive hai, aur sirf ek example use karna bahut noisy hota hai. Mini-batch GD ek compromise hai: ek choti random batch of examples se gradient estimate karo, ek step lo, repeat karo. Tum thodi si accuracy trade karte ho ek bahut bade speedup aur behtar hardware use ke liye.
Definition Full-batch objectiveHum N J ( θ ) = N 1 ∑ i = 1 N ℓ ( f θ ( x i ) , y i ) true gradient hai
∇ θ J ( θ ) = N 1 ∑ i = 1 N ∇ θ ℓ i
PROBLEM kya hai? N = 1 0 7 ∇ θ J ek bhi step lene se pehle . Shayad din mein kuch dozen updates hi milein. Learning bahut slow ho jaati.
Key statistical insight: true gradient ek average hai. Kisi bhi average ko ek random sample se estimate kiya ja sakta hai. Toh ek random subset B m ≪ N
g B = m 1 ∑ i ∈ B ∇ θ ℓ i
unbiased kyun hota hai
Agar B E [ ∇ θ ℓ i ] = N 1 ∑ j ∇ θ ℓ j = ∇ θ J m E [ g B ] = ∇ θ J ( θ ) average par hum sahi direction mein step karte hain — noise bahut saare steps mein cancel ho jaata hai.
Maano g i = ∇ θ ℓ i μ = ∇ J σ 2 g B = m 1 ∑ i = 1 m g i
Yeh step kyun? Hum jaanna chahte hain ki hamara estimate kitna spread hai, yeh batch size m
Var ( g B ) = Var ( m 1 ∑ i = 1 m g i ) = m 2 1 ∑ i = 1 m Var ( g i ) = m 2 1 ⋅ m σ 2 = m σ 2
Update rule (plain SGD form):
θ t + 1 = θ t − η g B t η epoch .
Definition Batch size method define karta hai
m = N Batch (full) gradient descent : exact gradient, smooth path, slow, memory-heavy.m = 1 Stochastic gradient descent (SGD) : bahut noisy, per step sasta, sharp minima se nikal sakta hai lekin optimum ke paas jhatkaa khaata hai.1 < m ≪ N Mini-batch gradient descent : practical sweet spot. Typical m ∈ { 32 , 64 , 128 , 256 }
feature kyun ho sakta hai
g B shallow/sharp minima aur saddle points se bahar koodne mein help karta hai, aur empirically solutions ko flatter minima ki taraf dhakelta hai jo better generalise karte hain. Full-batch GD, noiseless hone ki wajah se, jaahan land kare wahin stuck ho sakta hai.
Worked example 1 — Ek epoch mein updates count karnaDataset N = 50 , 000 m = 128 = ⌈ N / m ⌉ = ⌈ 50000/128 ⌉ = ⌈ 390.6 ⌉ = 391 Yeh step kyun? Aakhri batch 50000 − 390 × 128 = 80 remainder hai — tum phir bhi ise use karte ho (ya drop_last=True hone par drop karte ho). Toh 391 updates per epoch vs full-batch ke liye 1 update. 10 epochs mein: 3910 updates vs 10.
Worked example 3 — Feynman-style: expectation sahi haiData losses ke gradients hain [ 2 , − 4 , 6 , 0 ] = 4 2 − 4 + 6 + 0 = 1 { x 1 , x 3 } g B = 2 2 + 6 = 4 { x 2 , x 4 } g B = 2 − 4 + 0 = − 2 Yeh step kyun? Har ek batch galat hai (4, phir −2), lekin saare equally-likely size-2 batches ka average bilkul 1 ensemble honest hota hai.
Common mistake "Bada batch hamesha better train karta hai."Kyun sahi lagta hai: bada m Fix: noise sirf 1/ m aur helpful regularising noise gayab ho jaata hai, jo often generalisation hurt karta hai (sharp-minima problem). Compute double ho jaata hai lekin progress-per-FLOP girta hai. Sweet spot chhota–moderate m
Common mistake "Shuffling matter nahi karta."Kyun sahi lagta hai: data toh same hi hai dono taraf.
Fix: agar shuffle nahi karte, toh har epoch same batches same order mein dekhta hai → correlated, biased gradient sequence, aur agar data class ke hisaab se sorted hai toh model ek baar mein ek class dekhta hai. Har epoch shuffle karo taaki batches approximately i.i.d. rahein aur unbiasedness argument hold kare.
Common mistake "Mujhe batch size change karte waqt learning rate scale nahi karna chahiye."Kyun sahi lagta hai: learning rate alag knob lagta hai.
Fix: bada batch lower-variance gradient deta hai, toh tum safely bada step le sakte ho. Linear scaling rule: agar m k η ≈ k
Common mistake "Loss har single step mein neeche jaana chahiye."Kyun sahi lagta hai: GD ek descent method hai.
Fix: mini-batch loss noisy hota hai — individual steps loss badha sakte hain kyunki batch gradient true gradient nahi hai. Progress judge karo running/epoch average se, ek step se nahi.
Recall Active-recall checkpoints (answers chhupao, pehle try karo)
Mini-batch gradient estimator likho. → g B = m 1 ∑ i ∈ B ∇ θ ℓ i
Kya yeh biased hai? → Nahi, E [ g B ] = ∇ J
Iska std m σ / m
N = 1000 , m = 100 10 m = 1
Recall Feynman: 12-saal ke bachche ko explain karoSocho tum ek bahut bade school mein sabki average height jaanna chahte ho. 5000 sab bachon ko measure karna forever le lega. Iske bajaaye tum randomly 30 ka ek muthi pakad lete ho, unki heights average karte ho, aur use achha guess maante ho. Perfect nahi hai, lekin ek naye muthi se dobara karo aur usually close hota hai. Neural network train karna same hai: har photo dekhne se pehle network nudge karne ki bajaaye, tum photos ka ek chhota random dheer dekhte ho, sochte ho kidhar nudge karna hai, aur nudge karte ho. Chote dheere = bahut saare quick nudges. Kabhi kabhi ek dheera odd hota hai aur galat direction mein nudge karta hai, lekin hazaaron dheeron mein tum sahi direction mein drift karte ho. Yahi mini-batch gradient descent hai.
Mnemonic Trade-offs yaad karo"Batch is Slow-and-Sure, Stochastic is Fast-and-Furious, Mini is Just-Right."
Aur noise ke liye: "root-m rules" — noise 1/ m
Mini-batch GD kaunsa objective minimize karta hai? Dataset par average loss J ( θ ) = N 1 ∑ i ℓ i
Mini-batch gradient estimator kya hai? g B = m 1 ∑ i ∈ B ∇ θ ℓ i m
Kya mini-batch gradient unbiased hai? Haan — uniformly random batches ke liye, E [ g B ] = ∇ θ J
Gradient estimate ka standard deviation batch size m σ / m ki tarah (variance
σ 2 / m ).
Batch size chaar guna karna noise reduce karne ke liye inefficient kyun hai? Noise sirf aadha hota hai (
1/ m ) jabki cost chaar guna ho jaati hai — diminishing returns.
Epoch define karo. Saare training examples par ek full pass (saare mini-batches ek baar).
Steps per epoch formula? ⌈ N / m ⌉
Gradient noise generalisation kyun improve kar sakta hai? Yeh sharp minima aur saddle points se escape karne mein help karta hai, flatter minima ki taraf bias karta hai jo better generalise karte hain.
Batch size ke hisaab se teen regimes kya hain? m = N m = 1 1 < m ≪ N
Linear scaling rule kya hai? Jab batch size k k
Har epoch data shuffle karna kyun zaroori hai? Batches ko approximately i.i.d. rakhne ke liye taaki unbiased-gradient assumption hold kare aur correlated/class-ordered batches se bachne ke liye.
Mini-batch loss ek single step mein kyun badh sakta hai? Batch gradient ek noisy estimate hai; individual steps true loss descend nahi kar sakte bhale hi average trend kare.
true gradient is an average
exact computation over N is costly
estimate average from random sample
one pass over all batches
1 less than m less than N
Full-batch objective J theta
Update theta minus eta g_B
