Dropout regularization: Exercises — AI-ML deep dive

Everything below rests on three quantities from the parent note:

Here $r_{i} \sim Bernoulli (q)$ just means: flip a biased coin that lands "1" (keep) with probability $q$ and "0" (drop) with probability $p$ . See Bernoulli Distribution if that coin is new to you.

Level 1 — Recognition

Exercise 1.1

A framework documentation says Dropout(p=0.3). Following the PyTorch / original-paper convention, what fraction of neurons are kept on each forward pass, on average?

Recall Solution

p is the drop probability, so $p = 0.3$ means $30%$ dropped. $q = 1 - p = 1 - 0.3 = 0.7$ So on average $70%$ of neurons are kept. What we did: translated the label into keep-probability using $q = 1 - p$ . Why: frameworks name p as drop, and every scaling formula uses $q$ .

Exercise 1.2

True or false: dropout is active during test-time inference.

Recall Solution

False. Dropout is a training-only technique. At test time every neuron is used, deterministically. With inverted dropout the scaling was already applied during training, so the test forward pass is a plain forward pass.

Level 2 — Application

Exercise 2.1

A hidden layer outputs $a = [10, 20, 30, 40]$ . You use inverted dropout with $p = 0.25$ . The sampled mask this step is $r = [1, 1, 0, 1]$ . Compute the layer's output $\tilde{a}$ .

Recall Solution

Step 1 — keep probability. $q = 1 - 0.25 = 0.75$ , so the scale factor is $\frac{1}{q} = \frac{1}{0.75} = \frac{4}{3} \approx 1.3333$ . Step 2 — apply mask (elementwise multiply $r ⊙ a$ ): $[1, 1, 0, 1] ⊙ [10, 20, 30, 40] = [10, 20, 0, 40]$ Step 3 — rescale survivors by $4/3$ : $\tilde{a} = [\frac{4}{3} \cdot 10, \frac{4}{3} \cdot 20, 0, \frac{4}{3} \cdot 40] = [13.333, 26.667, 0, 53.333]$ Why the boost: dropping shrinks the expected signal to $q \cdot a$ ; multiplying survivors by $1/ q$ cancels that shrink so the expected output equals the original $a$ .

Exercise 2.2

For a single neuron with activation $a = 8$ under inverted dropout with $p = 0.5$ , list the two possible values of $\tilde{a}$ and the probability of each. Then compute $E [\tilde{a}]$ and confirm it equals $a$ .

Recall Solution

$q = 1 - 0.5 = 0.5$ , scale $= 1/ q = 2$ .

Neuron kept ( $r = 1$ ), probability $q = 0.5$ : $\tilde{a} = 2 \cdot 8 = 16$ .
Neuron dropped ( $r = 0$ ), probability $p = 0.5$ : $\tilde{a} = 0$ .

Expectation: $E [\tilde{a}] = q \cdot \frac{1}{q} a + p \cdot 0 = 0.5 \cdot 16 + 0.5 \cdot 0 = 8 = a ✓$ The $1/ q$ scaling is exactly what makes the average land back on the un-dropped value.

Level 3 — Analysis

Exercise 3.1

A layer has $n = 8$ dropable units. (a) How many distinct thinned sub-networks can dropout sample? (b) What is the probability that a specific sub-network (a specific on/off pattern) is chosen on one step, if $p = 0.5$ ?

Recall Solution

(a) Each unit is independently ON or OFF — 2 states. With $n$ independent units the number of patterns is $2^{n} = 2^{8} = 256.$ This is the " $2^{n}$ sub-networks" figure that makes dropout an approximate ensemble. (b) With $p = 0.5$ each unit is ON or OFF with probability $0.5$ , and the units are independent, so every one of the $256$ patterns is equally likely: $Pr [specific pattern] = (\frac{1}{2})^{8} = \frac{1}{256} \approx 0.0039.$ Note: equal probability only holds at $p = 0.5$ . For other $p$ a pattern with $k$ kept units has probability $q^{k} p^{n - k}$ , which varies across patterns.

Exercise 3.2

Show that under inverted dropout the variance injected into a single activation grows as the drop probability increases. Compute $Var [\tilde{a}]$ for a fixed activation $a$ , then evaluate it at $p = 0.2$ and $p = 0.5$ with $a = 1$ , and state which is noisier.

Recall Solution

Write $\tilde{a} = \frac{r}{q} a$ with $r \sim Bernoulli (q)$ . A Bernoulli( $q$ ) variable has $Var [r] = q (1 - q) = q p$ . Since $\frac{a}{q}$ is a constant multiplier, pull it out (variance scales by the square of a constant): $Var [\tilde{a}] = (\frac{a}{q})^{2} Var [r] = \frac{a ^{2}}{q ^{2}} q p = a^{2} \frac{p}{q} = a^{2} \frac{p}{1 - p} .$ Why $p / q$ : the ratio $p / (1 - p)$ is small when $p$ is small and blows up as $p \to 1$ — more dropping, more noise. With $a = 1$ :

$p = 0.2$ : $\frac{0.2}{0.8} = 0.25$ .
$p = 0.5$ : $\frac{0.5}{0.5} = 1.0$ .

So $p = 0.5$ injects 4× the variance of $p = 0.2$ — it is noisier. This noise is exactly the "smoother function" mechanism from the parent note, and it is why input layers use a smaller $p$ (you don't want to corrupt the raw data too hard).

Level 4 — Synthesis

Exercise 4.1

You are told a network trained without scaling: it used the original (non-inverted) convention, meaning masking with no train-time boost, and a test-time scale-down. Its layer produces raw test-time activation $a = 12$ for some unit, and it was trained with $p = 0.4$ . What test-time output should this unit produce, and why is inverted dropout usually preferred over doing this?

Recall Solution

Original convention: no scaling at train time, so during training the next layer saw an expected input of $q \cdot a$ . To match that at test time (all units on), you must scale down by $q$ : $q = 1 - 0.4 = 0.6, \tilde{a}_{test} = q \cdot a = 0.6 \cdot 12 = 7.2.$ Why inverted is preferred: inverted dropout puts the $1/ q$ scaling into the training path, leaving the test / inference forward pass as a plain, unmodified pass. That means deployment code is simpler and you cannot forget the correction at inference time — the most error-prone moment.

Exercise 4.2

Design choice. You have a 3-layer MLP overfitting badly (train accuracy $99%$ , val accuracy $80%$ — a large variance gap). You must pick dropout rates for: (i) the input layer, (ii) the two hidden layers. Choose sensible $p$ values, justify each, and explain how you'd expect the generalization gap to move.

Recall Solution

(i) Input layer: small drop, e.g. $p = 0.1$ – $0.2$ . Reason: inputs are the raw signal; dropping too many corrupts the data itself (recall from 3.2 that $Var [\tilde{a}] = a^{2} p / q$ grows fast with $p$ ). Pick $p_{in} = 0.1$ . (ii) Hidden layers: stronger drop, e.g. $p = 0.5$ . Reason: this is where co-adaptation and memorisation live; $p = 0.5$ maximises the number of effectively-explored sub-networks and injects the most regularising noise. Pick $p_{hidden} = 0.5$ . Expected effect: train accuracy will drop (the model is now handicapped during training — good, it means less memorisation), while val accuracy rises, shrinking the $99% - 80% = 19$ -point gap. This is trading a little bias for a large variance reduction — the exact bargain weight decay and dropout both offer.

Level 5 — Mastery

Exercise 5.1

Ensemble equivalence, worked exactly. Consider a linear unit with two inputs and no bias: output $y = w_{1} x_{1} + w_{2} x_{2}$ . Apply independent dropout with keep probability $q$ to each input using inverted dropout, so the actual output is $\tilde{y} = w_{1} \frac{r _{1}}{q} x_{1} + w_{2} \frac{r _{2}}{q} x_{2}, r_{1}, r_{2} \sim Bernoulli (q) independent .$ Show that $E [\tilde{y}] = y$ (dropout is unbiased for a linear unit), i.e. the ensemble's average prediction equals the full-network prediction. Verify numerically with $w_{1} = 3, w_{2} = 5, x_{1} = 2, x_{2} = 4, q = 0.5$ by averaging over all four masks.

Recall Solution

Symbolic. Expectation is linear, and $E [r_{i}] = q$ : $E [\tilde{y}] = w_{1} \frac{E [ r _{1} ]}{q} x_{1} + w_{2} \frac{E [ r _{2} ]}{q} x_{2} = w_{1} \frac{q}{q} x_{1} + w_{2} \frac{q}{q} x_{2} = w_{1} x_{1} + w_{2} x_{2} = y .$ So the inverted-dropout output is an unbiased estimate of the full output — the average over sub-networks is the full network, exactly, for a linear unit. (For nonlinear units it is only an approximation, which is why test-time scaling is called approximate ensembling.)

Numeric check ( $q = 0.5$ , scale $1/ q = 2$ ). Full output: $y = 3 \cdot 2 + 5 \cdot 4 = 6 + 20 = 26$ . The four equally-likely masks $(r_{1}, r_{2})$ give:

$(1, 1)$ : $2 (3 \cdot 2) + 2 (5 \cdot 4) = 12 + 40 = 52$
$(1, 0)$ : $2 (3 \cdot 2) + 0 = 12$
$(0, 1)$ : $0 + 2 (5 \cdot 4) = 40$
$(0, 0)$ : $0$

Average $= \frac{52 + 12 + 40 + 0}{4} = \frac{104}{4} = 26 = y ✓$ . The ensemble mean lands exactly on the full-network prediction.

Exercise 5.2

Effective learning-rate / weight-scaling insight. A common claim: "for a linear neuron, dropout on the input behaves like an $L_{2}$ -style penalty on the weights that scales with $p / q$ ." Using the variance result $Var [\tilde{a}] = a^{2} p / q$ from Exercise 3.2, argue heuristically why larger $p$ acts like a stronger weight penalty, and compute the relative penalty strength ratio between $p = 0.5$ and $p = 0.1$ .

Recall Solution

Heuristic. The squared-error loss on a noisy input has an expected value that splits into (loss on the clean signal) $+$ (a term proportional to the injected variance). For a linear unit that variance term is $\propto w^{2} Var [\tilde{x}] = w^{2} x^{2} \frac{p}{q}$ . Minimising it pushes $w^{2}$ down — precisely what an $L_{2}$ / weight-decay penalty does. The penalty strength is governed by the factor $\frac{p}{q} = \frac{p}{1 - p}$ : bigger $p$ ⇒ bigger factor ⇒ stronger implicit shrinkage. Ratio. $\frac{( p / q ) _{p = 0.5}}{( p / q ) _{p = 0.1}} = \frac{0.5/0.5}{0.1/0.9} = \frac{1.0}{0.111 \dots} = 9.$ So $p = 0.5$ imposes roughly a 9× stronger implicit weight penalty than $p = 0.1$ . This quantifies why hidden layers (want strong regularisation) use $p \approx 0.5$ while inputs (want gentle regularisation) use $p \approx 0.1$ .

Active Recall

Question: Scale factor for surviving activations in inverted dropout

\frac{1}{q} = \frac{1}{1 - p}

Question: Number of sub-networks for

n

dropable units

2^{n}

, equally likely only when

p = 0.5

Question: Injected variance of a single activation

Var [\tilde{a}] = a^{2} \frac{p}{q}

, exploding as

p \to 1

Question: Is dropout exactly ensemble-equivalent for deep nets

Only for linear units; nonlinearities make it an approximation.

Connections

Overfitting and Generalization — every exercise here targets shrinking the generalization gap.
Ensemble Methods (Bagging) — Exercises 3.1 and 5.1 make the ensemble view concrete.
L2 Regularization (Weight Decay) — Exercise 5.2 links dropout to an implicit weight penalty.
Bias-Variance Tradeoff — Exercise 4.2 is a pure bias-variance trade in action.
Bernoulli Distribution — the mask $r_{i}$ throughout.
Batch Normalization — an alternative regulariser worth comparing when tuning dropout.
🇮🇳 Hinglish version of the parent note.