Foundations — Dropout regularization

This page assumes nothing. Before you touch the parent note Dropout regularization, we build every symbol it uses, in the order they depend on each other. If a term below already feels obvious, skim it — but the later math only clicks if each of these pictures is in your head.

0. What is a neuron and an "activation"? (the thing being dropped)

Before we can drop anything, we need to know what gets dropped.

Look at the figure. Each circle is one neuron; the number floating next to it is its activation — how strongly that neuron "fired" for this input. Dropout does one brutal thing to this picture: it reaches in and forces some of these numbers to zero.

• Why the topic needs this: dropout is applied to activations, not to the weights (the connection strengths). Keep that straight — we silence outputs, not connections.

1. The symbol $a$ and $a_i$ — a layer's activations

Why the topic needs it: every dropout formula acts on one neuron at a time ($a_i$), then we bundle them back into the whole layer ($a$).

2. Probability — the language of "randomly"

Dropout is random, so we need a precise way to say "how often". That precise way is probability.

The bar in the figure is split: the shaded slice is the chance of one outcome, the rest is the chance of the other. The two slices always add to $1$, because something must happen every time.

Why the topic needs it: "each neuron is dropped with probability $p$" is meaningless until you know a probability is just a fraction-of-the-time between $0$ and $1$.

3. The symbols $p$ and $q$ — drop chance and keep chance

Now we name the two probabilities dropout cares about.

Because a neuron must either be dropped or kept (nothing else can happen), their probabilities fill the whole bar:

$p+q=1⟺q=1-p$

Why the topic needs it: every later formula uses $q$ as the shrink/boost factor. $p$ sets the amount of randomness; $q$ is what you multiply and divide by.

4. The Bernoulli distribution and the mask $r$

We need a mathematical "coin" that outputs $1$ (keep) or $0$ (drop). That coin is the Bernoulli distribution.

The whole row of coins is the mask $r$: one coin per neuron.

In the figure, each neuron gets its own coin. Where the coin shows $1$ (mint), the activation passes through. Where it shows $0$ (coral), the activation is crushed to zero. That is the entire dropping mechanism:

${\tilde{a}}_i=r_i{a}_i$

Read this as: "the dropped activation ${\tilde{a}}_i$ (say 'ay-tilde') equals the coin $r_i$ times the original $a_i$." If $r_i=1$, nothing changes; if $r_i=0$, the neuron is silenced.

• The tilde $\tilde{a}$ just means "the modified version of $a$" — a hat that says this has been messed with by dropout.
• "Independently at every forward pass" means each neuron flips its own coin, and fresh coins are flipped every step. Neighbours don't share coins.

Why the topic needs it: the mask $r$ is the actual object dropout multiplies your layer by. See Bernoulli Distribution and, for why $2^n$ different masks give an ensemble, Ensemble Methods (Bagging).

5. Expectation $\mathbb{E}[\cdot ]$ — the "long-run average"

The most confusing symbol in the parent note is $\mathbb{E}$. It is simple.

For a Bernoulli($q$) coin, the average of "$1$ with chance $q$, $0$ with chance $p$" is:

$\mathbb{E}[r_i]=q\cdot 1+p\cdot 0=q$

A key rule we will lean on: constants slide out of expectation. Since $a_i$ is a fixed number (not random) once the input is fixed,

$\mathbb{E}[r_i{a}_i]=\mathbb{E}[r_i]a_i=q{a}_i$

Why the topic needs it: dropping shrinks the average signal to $q{a}_i$. That shrink is the whole reason the $1/q$ rescale exists — which is the next symbol.

6. Rescaling by $1/q$ — restoring the honest average

Here is the problem the parent note solves. During training the next layer sees signals shrunk to $q{a}_i$ on average. At test time we keep everyone, so it suddenly sees the full $a_i$ — which is $1/q$ times bigger. Mismatch.

The fix (inverted dropout) is to boost survivors during training so the average never shrinks:

${\tilde{a}}_i=\frac{r_i}{q}{a}_i$

Check the average is now honest:

$\mathbb{E}\left[\frac{r_i}{q}{a}_i\right]=\frac{1}{q}\mathbb{E}[r_i]a_i=\frac{1}{q}\cdot q\cdot a_i=a_i$

The $q$'s cancel — the expected activation equals the original. No test-time change needed.

Why the topic needs it: without this factor, test activations are $1/q$ too large and the model performs worse than using no dropout at all.

Prerequisite map

Equipment checklist

Cover the right side and test yourself — if any answer is fuzzy, reread its section above.

Connections

• Bernoulli Distribution — the coin behind the mask $r$.
• Overfitting and Generalization — why we bother switching neurons off at all.
• Parent: Dropout regularization — where these symbols get used in full.
• Ensemble Methods (Bagging) — the $2^n$ sub-networks the mask secretly creates.