Dropout regularization: Worked examples — AI-ML deep dive

This deep dive drills the arithmetic of dropout until no case can surprise you. The parent note built the ideas; here we exhaust the numbers. Before we compute anything, let us name every symbol so nothing appears unearned.

Definition The symbols we will use (all from the parent, restated)

$a$ — a vector of activations, the raw outputs a layer produces before dropout. Picture a row of light bulbs, each with a brightness number.
$p$ — the drop probability: the chance that any one bulb is switched off this step. A number between $0$ and $1$ .
$q = 1 - p$ — the keep probability: the chance a bulb stays on.
$r$ — the mask, a vector of $0$ s and $1$ s. $r_{i} = 1$ means "bulb $i$ kept on", $r_{i} = 0$ means "bulb $i$ switched off". Each $r_{i}$ is drawn from a Bernoulli Distribution: it is $1$ with probability $q$ and $0$ with probability $p$ .
$⊙$ — elementwise multiply: multiply matching positions, $[a, b] ⊙ [c, d] = [a c, b d]$ . We use it because the mask hits each neuron independently.
$E [\cdot]$ — expectation, the long-run average over infinitely many random masks. If a bulb is on half the time at brightness $6$ , its expected brightness is $3$ .
$\tilde{a}$ — the activation after dropout (and scaling), the thing actually sent forward.

The scenario matrix

Every question dropout can throw at you falls into one of these cells. The examples below are labelled by which cell they hit, and together they cover all of them.

Cell	Scenario class	What could go wrong
A	Standard train-time mask + rescale	forgetting the $1/ q$ boost
B	Test-time forward pass	accidentally masking at test
C	Extreme $p$ : $p \to 0$ (keep all)	scale factor $\to 1$ , dropout vanishes
D	Extreme $p$ : $p \to 1$ (drop all)	division by $q \to 0$ blows up
E	Degenerate mask: all-drop / all-keep sample	a single unlucky step
F	Expectation over many masks	proving the average is preserved
G	Original (non-inverted) convention	scale DOWN at test by $q$
H	Real-world word problem	choosing $p$ , reading a framework
I	Exam twist: variance, not just mean	dropout changes spread, not average
J	Ensemble counting ( $2^{n}$ subnets)	miscounting sub-networks

The figure above shows Cell A happening: raw bulbs on the left, a random on/off mask in the middle, and the boosted survivors on the right. Keep it in view for Example 1.

Example 1 — Cell A: standard train-time step

Forecast: guess before reading — will the surviving numbers be bigger or smaller than $2$ and $6$ ?

Apply the mask. $r ⊙ a = [1 \cdot 2, 0 \cdot 4, 1 \cdot 6, 0 \cdot 8] = [2, 0, 6, 0]$ . Why this step? $r_{i} = 0$ literally switches a bulb off; multiplication by $0$ is how "off" is encoded.
Rescale survivors by $1/ q = 1/0.5 = 2$ . $\tilde{a} = 2 \cdot [2, 0, 6, 0] = [4, 0, 12, 0]$ . Why this step? On average only half the bulbs survive, so each survivor must shine twice as bright to keep the layer's total expected signal unchanged.

Verify: expected value of position 1 over all masks $= q \cdot \frac{1}{q} \cdot 2 = 2$ ✅ (matches original). The survivors got bigger — forecast confirmed.

Example 2 — Cell B: the test-time pass

Forecast: more zeros? Bigger numbers? Or exactly $a$ ?

No mask at test. With inverted dropout, test uses $\tilde{a} = a$ . Why this step? All the scaling was already paid during training (the $1/ q$ boost), so inference is a plain deterministic forward pass.
Output: $\tilde{a} = [2, 4, 6, 8]$ . Why this step? Every bulb is on; no random $r$ exists at test.

Verify: the average train output at each position was also $a_{i}$ (Example 1's check). Train-average equals test value $\Rightarrow$ no distribution shift. ✅

Example 3 — Cell C: the limit $p \to 0$

Forecast: does the boost matter here?

Scale factor: $1/ q = 1/ (1 - 0.01) = 1/0.99 \approx 1.0101$ . Why this step? Almost all bulbs survive, so the correction is tiny.
Apply: $\tilde{a} = 1.0101 \cdot [10, 20] = [10.101, 20.202]$ .
Limit: as $p \to 0$ , $q \to 1$ , $1/ q \to 1$ , so $\tilde{a} \to a$ . Dropout disappears. Why this step? If you never drop anyone, there is nothing to correct — regularization strength goes to zero.

Verify: $1/0.99 = 1.0101 \dots$ and $lim_{p \to 0} 1/ (1 - p) = 1$ . ✅

Example 4 — Cell D: the limit $p \to 1$

Forecast: what does $1/ q$ do as $q$ shrinks toward $0$ ?

Scale factor: $1/ q = 1/ (1 - 0.999) = 1/0.001 = 1000$ . Why this step? Almost every bulb is off, so the rare survivor must be boosted enormously to represent the whole layer.
A surviving neuron at value $5$ becomes $5000$ — a huge, noisy spike. Why this step? One survivor now carries the signal of the entire layer, so its rescaled value explodes.
At exactly $p = 1$ : $q = 0$ , and $1/ q$ is division by zero — undefined. Also every bulb is off, so there is literally no signal. Why this step? The formula and the concept both break; you must keep $p < 1$ .

Verify: $1/ (1 - 0.999) = 1000$ exactly, and $q = 0$ makes $1/ q$ undefined. ✅

Example 5 — Cell E: an unlucky degenerate sample

Forecast: how likely is each, and is either "wrong"?

All dropped: $\tilde{a} = \frac{1}{q} [0, 0, 0, 0] = [0, 0, 0, 0]$ . Probability $= p^{4} = 0. 5^{4} = 0.0625$ . Why this step? Each independent bulb is off with prob $p$ ; all four off $= p^{4}$ .
All kept: $\tilde{a} = \frac{1}{0.5} [3, 3, 3, 3] = [6, 6, 6, 6]$ . Probability $= q^{4} = 0. 5^{4} = 0.0625$ . Why this step? All four on $= q^{4}$ ; survivors doubled by $1/ q$ .
Interpretation: neither single step is "wrong" — dropout is deliberately noisy. Correctness lives in the average, not any one sample.

Verify: $0. 5^{4} = 0.0625$ for both. ✅

Example 6 — Cell F: expectation over many masks

Forecast: will the average land above, below, or exactly on $a$ ?

One position's expectation: $E [\tilde{a}_{i}] = E [\frac{r _{i}}{q} a_{i}] = \frac{a _{i}}{q} E [r_{i}]$ . Why this step? $a_{i}$ and $q$ are constants; only $r_{i}$ is random, so pull constants out.
Insert $E [r_{i}] = q$ (Bernoulli mean): $= \frac{a _{i}}{q} \cdot q = a_{i}$ . Why this step? This is the whole point of the $1/ q$ factor — it exactly cancels the shrink.
Vector result: $E [\tilde{a}] = [2, 4, 6, 8] = a$ .

Verify (numeric simulation intent): for position with $a_{i} = 8$ , average $= q \cdot \frac{1}{q} \cdot 8 = 8$ . ✅ Forecast: exactly on $a$ .

Example 7 — Cell G: original (non-inverted) convention

Forecast: which convention puts the extra work at test time?

Train (no scale): $\tilde{a} = r ⊙ a = [2, 0, 6, 0]$ . Why this step? Original dropout leaves survivors at raw brightness during training.
Test (scale down by $q$ ): $\tilde{a} = q \cdot a = 0.5 \cdot [2, 4, 6, 8] = [1, 2, 3, 4]$ . Why this step? At test all bulbs are on, so their raw sum is $1/ q$ too big; multiply by $q$ to shrink back to training scale.

Verify: train-time expected survivor value $= q \cdot a_{i}$ ; test value $= q \cdot a_{i}$ . Both equal $q a_{i}$ ⇒ matched. For $a_{i} = 8$ : $0.5 \cdot 8 = 4$ . ✅ (Inverted moves the work to train; original moves it to test.)

Example 8 — Cell H: real-world word problem

Forecast: is p closer to $0.3$ or $0.7$ ?

Keep fraction wanted: $q = 358/512 = 0.699 \dots \approx 0.7$ . Why this step? Expected active units $= q \times (total)$ , so $q$ is the target keep fraction.
Convert to PyTorch p: $p = 1 - q = 1 - 0.7 = 0.3$ . Why this step? PyTorch's nn.Dropout(p) treats p as drop probability (a classic trap — see the parent's mistakes).
Expected active units: $E [active] = q \cdot 512 = 0.7 \cdot 512 = 358.4$ . Why this step? Sum of $512$ independent Bernoulli( $q$ ) keeps has mean $512 q$ .

Verify: $0.7 \times 512 = 358.4$ , and $p = 1 - 0.7 = 0.3$ . ✅ Forecast: closer to $0.3$ .

Example 9 — Cell I: exam twist (variance, not mean)

Forecast: is the variance zero (mean is preserved!) or positive?

Possible values: $\tilde{a} = \frac{1}{q} a = 20$ with prob $q = 0.5$ , or $0$ with prob $p = 0.5$ . Why this step? $r$ is either $1$ (survive, boosted to $20$ ) or $0$ (dropped to $0$ ).
Mean: $0.5 \cdot 20 + 0.5 \cdot 0 = 10$ (matches, good).
Variance $= E [\tilde{a}^{2}] - (E [\tilde{a}])^{2}$ . Here $E [\tilde{a}^{2}] = 0.5 \cdot 2 0^{2} + 0.5 \cdot 0^{2} = 200$ . Why this step? Variance measures spread; we need the average of squares minus square of average.
Result: $Var = 200 - 1 0^{2} = 100$ . Why this step? This positive spread is the regularizing noise — the mean stays honest while the signal jitters.

Verify: $0.5 \cdot 400 - 100 = 100$ . General formula: $Var = a^{2} \frac{p}{q} = 100 \cdot \frac{0.5}{0.5} = 100$ . ✅ Forecast: positive.

Example 10 — Cell J: counting the ensemble

Forecast: does adding one unit add or double the sub-network count?

Each unit is a switch with $2$ states (on/off), chosen independently. Why this step? A sub-network is fully described by which units are on; that's a length- $n$ binary string.
Total configurations $= 2^{n}$ . For $n = 10$ : $2^{10} = 1024$ . For $n = 3$ : $2^{3} = 8$ . Why this step? Independent binary choices multiply: $2 \times 2 \times \dots$ ( $n$ times).
Ensemble link: dropout trains one random member of this $2^{n}$ -family per step; test-time weight-sharing approximates averaging all of them — a free bagging ensemble.

Verify: $2^{10} = 1024$ , $2^{3} = 8$ ; adding one unit doubles the count ( $2^{n + 1} = 2 \cdot 2^{n}$ ). ✅ Forecast: doubles.

Active Recall

Connections

3.2.10 Dropout regularization (Hinglish) — the parent topic these examples drill.
Bernoulli Distribution — every mask entry $r_{i}$ is Bernoulli( $q$ ); its mean $q$ drives all the arithmetic here.
Overfitting and Generalization — Example 8's word problem is exactly closing the generalization gap.
Ensemble Methods (Bagging) — the $2^{n}$ count of Example 10.
Bias-Variance Tradeoff — the added variance of Example 9 is the price paid for lower variance in the final model.
L2 Regularization (Weight Decay) — an alternative regularizer to compare scale choices against.
Batch Normalization — often reduces how aggressive $p$ needs to be.

Worked examples — Dropout regularization

The scenario matrix

Example 1 — Cell A: standard train-time step

Example 2 — Cell B: the test-time pass

Example 3 — Cell C: the limit $p \to 0$

Example 4 — Cell D: the limit $p \to 1$

Example 5 — Cell E: an unlucky degenerate sample

Example 6 — Cell F: expectation over many masks

Example 7 — Cell G: original (non-inverted) convention

Example 8 — Cell H: real-world word problem

Example 9 — Cell I: exam twist (variance, not mean)

Example 10 — Cell J: counting the ensemble

Active Recall

Connections

Scenario Map