Dropout regularization

3.2.10 · AI-ML › Training Deep Networks

WHAT is Dropout?

$p$ = probability ki ek neuron drop hoga (typical: hidden ke liye $0.5$ , input ke liye $0.1$ – $0.2$ ).
$q = 1 - p$ = probability ki ek neuron kept rahega.
Dropout activations pe apply hota hai, weights pe nahi.

HOW: the math, derived from scratch

Training-time forward pass

Ek layer output $a$ ke liye, ek mask vector $r$ define karo jahan har component yeh hai:

$r_{i} \sim Bernoulli (q), q = 1 - p$

Dropped activation yeh hai:

$\tilde{a}_{i} = r_{i} a_{i}$

Sirf surviving neurons ( $r_{i} = 1$ ) aage pass hote hain; baaki sab zero ho jaate hain.

The scaling problem — derived

Scaling ki zaroorat kyun hai? Ek single dropped activation ki expectation dekho:

$E [\tilde{a}_{i}] = E [r_{i}] a_{i} = q a_{i}$

Yeh step kyun? $r_{i}$ Bernoulli( $q$ ) hai, toh $E [r_{i}] = q$ . Dropping expected signal ko $q$ factor se shrink kar deta hai.

Test time pe hum saare neurons rakhte hain, toh raw activation $a_{i}$ hai — jo training ke dauran next layer ne jo dekha tha usase $1/ q$ times bada hai. Next layer ke weights chhote training-scale input ke liye tune kiye gaye the, toh test activations inflated ho jaayenge → mismatch.

Fix (do equivalent conventions):

Inverted kyun preferred hai: test-time / inference code saaf rehta hai (bas ek normal forward pass), aur saara extra kaam sirf training ke dauran hota hai.

Worked Example 1 — one hidden layer

Maano ek hidden layer activations $a = [2, 4, 6, 8]$ produce karta hai aur hum inverted dropout use karte hain $p = 0.5$ ke saath toh $q = 0.5$ .

Is step pe sampled mask: $r = [1, 0, 1, 0]$ .

Step 1 — mask apply karo. $r ⊙ a = [2, 0, 6, 0]$ Kyun? Neurons 2 aur 4 drop ho gaye → force hokar 0 ho gaye.

Step 2 — $1/ q = 2$ se rescale karo. $\tilde{a} = [4, 0, 12, 0]$ Kyun? Average pe sirf aadhe neurons survive karte hain, toh surviving waalon ko $2 \times$ boost diya jaata hai taaki expected sum $[2, 4, 6, 8]$ jaisa hi rahe.

Expectation check karo kai masks ke upar: har $a_{i}$ prob $q = 0.5$ ke saath appear karta hai, $2$ se scale hokar, mean $0.5 \cdot 2 \cdot a_{i} = a_{i}$ deta hai. Expected activation = original. ✅

Worked Example 2 — why co-adaptation breaks

Do neurons $h_{1}, h_{2}$ milke "cat" detect karte hain sirf tabhi jab dono strongly fire karein (co-adapted).

Dropout ke bina: net seekhta hai $y = w_{1} h_{1} + w_{2} h_{2}$ jo sirf tabhi kaam karta hai jab dono present hon.
$p = 0.5$ ke saath: aadhe time $h_{2}$ drop ho jaata hai. Net ko phir bhi "cat" gradient milta hai, toh $h_{1}$ ko akele hi useful signal carry karna padta hai.

Yeh step kyun matter karta hai: random absences mein perform karne ka pressure redundant, robust features force karta hai — yahi regularization ka essence hai.

Common Mistakes

Recall Feynman: explain to a 12-year-old

Tum ek sports team mein ho aur coach randomly kuch players ko har practice pe bench karta hai. Kyunki tumhe pata nahi hota ki kaun missing hoga, sab ko acchi tarah khelna aur ek doosre ko cover karna seekhna padta hai. Game day (test time) pe sab khelते hain — aur team bahut zyada strong hoti hai kyunki koi bhi ek weak link nahi hota jis pe doosre secretly depend karte the. Dropout practice ke dauran random neurons ko bench karta hai taaki network kisi ek pe depend karna band kar de.

Active Recall

Training ke dauran dropout kya randomly zero set karta hai?

Neuron activations (outputs), har ek ko drop probability

p

ke saath, har forward pass pe independently.

Inverted dropout mein, train time pe surviving activations ko kaunse factor se scale kiya jaata hai?

1/ q = 1/ (1 - p)

se, taaki expected activation un-dropped value ke barabar rahe.

Scaling ki zaroorat kyun hai?

Kyunki dropping expected activation ko

q \cdot a

kar deta hai; correction ke bina, test-time (full) activations

1/ q

times badi hongi jo next layer ke liye train nahi ki gayi thi.

Kya test time pe dropout apply hota hai?

Nahi — test time pe saare neurons deterministically use hote hain; sirf scaling correction apply hoti hai (inverted dropout ke liye baked in hoti hai).

Network mein kaunsi problem ko dropout mainly attack karta hai?

Co-adaptation / overfitting — neurons ka specific doosre neurons pe over-rely karna taaki training noise fit ho sake.

n

dropable units ke upar dropout implicitly kitne sub-networks ensemble karta hai?

Lagbhag

2^{n}

possible thinned sub-networks.

Hidden layers aur input layers ke liye typical drop probability?

Hidden layers ke liye ~0.5, input layers ke liye ~0.1–0.2 (chhota).

E [r_{i} a_{i}]

jahan

r_{i} \sim Bernoulli (q)

q a_{i}

, kyunki

E [r_{i}] = q

Connections

Overfitting and Generalization — dropout generalization gap ko reduce karne ka ek tool hai.
L2 Regularization (Weight Decay) — alternative/complementary regularizer; dropout ek adaptive penalty ki tarah act karta hai.
Ensemble Methods (Bagging) — dropout ≈ sub-networks ke upar implicit bagging.
Batch Normalization — dropout ke saath interact karta hai; aksar BN iske need ko reduce kar deta hai.
Bernoulli Distribution — mask ka underlying probability model.
Bias-Variance Tradeoff — dropout thoda bias trade karta hai ek badi variance reduction ke liye.