Dropout regularization

WHAT is Dropout?

• $p$ = probability a neuron is dropped (typical: $0.5$ for hidden, $0.1$–$0.2$ for input).
• $q = 1-p$ = probability a neuron is kept.
• Dropout is applied to activations, not weights.

WHY does it work?

HOW: the math, derived from scratch

Training-time forward pass

For a layer output $a$, define a mask vector $r$ where each component is:

$r_i \sim \text{Bernoulli}(q), \qquad q = 1-p$

The dropped activation is:

$\tilde{a}_i = r_i \, a_i$

Only the surviving neurons ($r_i=1$) pass forward; the rest are zeroed.

The scaling problem — derived

Why do we need scaling? Look at the expectation of a single dropped activation:

$\mathbb{E}[\tilde a_i] = \mathbb{E}[r_i]\, a_i = q\, a_i$

Why this step? $r_i$ is Bernoulli($q$), so $\mathbb{E}[r_i] = q$. Dropping shrinks the expected signal by a factor $q$.

At test time we keep all neurons, so the raw activation is $a_i$ — which is $1/q$ times bigger than what the next layer saw during training. The next layer's weights were tuned to the smaller training-scale input, so test activations would be inflated → mismatch.

Fix (two equivalent conventions):

Why inverted is preferred: the test-time / inference code stays clean (just a normal forward pass), and all the extra work happens only during training.

Worked Example 1 — one hidden layer

Suppose a hidden layer produces activations $a = [2, 4, 6, 8]$ and we use inverted dropout with $p = 0.5$ so $q = 0.5$.

Sampled mask this step: $r = [1, 0, 1, 0]$.

Step 1 — apply mask. $r \odot a = [2, 0, 6, 0]$ Why? Neurons 2 and 4 were dropped → forced to 0.

Step 2 — rescale by $1/q = 2$. $\tilde a = [4, 0, 12, 0]$ Why? On average only half the neurons survive, so surviving ones are boosted $2\times$ to keep the expected sum the same as $[2,4,6,8]$.

Check the expectation over many masks: each $a_i$ appears with prob $q=0.5$, scaled by $2$, giving mean $0.5\cdot 2\cdot a_i = a_i$. Expected activation = original. ✅

Worked Example 2 — why co-adaptation breaks

Two neurons $h_1, h_2$ jointly detect "cat" only when both fire strongly (co-adapted).

• Without dropout: the net learns $y = w_1 h_1 + w_2 h_2$ that only works when both present.
• With $p=0.5$: half the time $h_2$ is dropped. The net still gets the "cat" gradient, so $h_1$ alone must carry useful signal.

Why this step matters: the pressure to perform under random absences forces redundant, robust features — the essence of the regularization.

Common Mistakes

Active Recall

Connections

• Overfitting and Generalization — dropout is a tool to reduce the generalization gap.
• L2 Regularization (Weight Decay) — alternative/complementary regularizer; dropout acts like an adaptive penalty.
• Ensemble Methods (Bagging) — dropout ≈ implicit bagging over sub-networks.
• Batch Normalization — interacts with dropout; often BN reduces the need for it.
• Bernoulli Distribution — the mask's underlying probability model.
• Bias-Variance Tradeoff — dropout trades a bit of bias for a large variance reduction.

Concept Map

Hinglish (regional understanding)