Foundations — Reparameterization trick

This page assumes you have seen none of the notation the Reparameterization trick parent note uses. We build every symbol from the ground up, in an order where each one leans only on the ones before it. Prefer Hindi-English? See the Hinglish version →.

1. What is a random variable? (the symbol $z$, and "$\sim$")

The squiggle $\sim$ is read "is drawn from" or "is distributed as". So

$z\sim \mathcal{N}(\mu ,{\sigma }^2)$

reads: "$z$ is a random number produced by the bell curve $\mathcal{N}$." We haven't defined $\mathcal{N}$, $\mu$, or $\sigma$ yet — that's next. For now, just picture a machine that spits out a fresh number each time you press a button.

Look at the figure: each press of the button drops a dot at a different spot along the number line, but the dots cluster. That clustering shape is exactly the distribution.

2. The bell curve: $\mathcal{N}$, $\mu$, $\sigma$, ${\sigma }^2$

The two numbers:

• $\mu$ (Greek letter "mu") = the mean = where the peak sits on the number line. Slide $\mu$ right, the whole bell slides right.
• $\sigma$ (Greek "sigma") = the standard deviation = how wide the bell is. Big $\sigma$ = fat, spread-out bell; small $\sigma$ = tall, narrow spike. We always require $\sigma >0$ (a width can't be negative or zero).
• ${\sigma }^2$ = "sigma squared" = the variance. It is just $\sigma$ multiplied by itself. We square it because widths add up naturally in squared units (this is a fact of statistics; take it as given here).

So the notation $\mathcal{N}(\mu ,{\sigma }^2)$ means "the bell curve centered at $\mu$ with width-squared ${\sigma }^2$."

3. Expected value $\mathbb{E}[\cdot ]$ and variance $\text{Var}(\cdot )$

Two facts we will lean on (both follow from the definitions above — "averaging is fair to shifting and scaling"):

• $\mathbb{E}[\mu +\sigma ϵ]=\mu +\sigma \mathbb{E}[ϵ]$ — you can pull constants out of an average.
• $\text{Var}(\mu +\sigma ϵ)={\sigma }^2\text{Var}(ϵ)$ — adding the constant $\mu$ shifts everything equally so spread is unchanged; scaling by $\sigma$ multiplies every distance-from-mean by $\sigma$, hence squared distances by ${\sigma }^2$.

A subscript on $\mathbb{E}$ tells you which random source you're averaging over:

${\mathbb{E}}_{ϵ\sim \mathcal{N}(0,1)}[f(ϵ)]$

means "average $f(ϵ)$ as $ϵ$ ranges over the standard normal." The subscript matters enormously later: the whole trick is about changing which variable the expectation is over.

4. A function of a random number: $f(z)$

Picture $f$ as a valley: the bottom of the valley is the best $z$, the walls punish bad values.

5. Slope of a function: the derivative $\frac{\partial f}{\partial z}$

Why we want it: the slope tells us which way is downhill. If the slope is negative, increasing $z$ lowers the loss; if positive, decreasing $z$ lowers it. That is the entire engine of learning.

6. The chain rule: gradients that pass through a middle step

The reparameterization trick exists precisely so that the middle piece $\frac{\partial z}{\partial \mu }$ actually exists. If $z$ is drawn by a random button, that middle slope is undefined and the chain snaps. If $z$ is a formula, the middle slope is a clean number and the chain holds.

7. The parameters $\varphi$, the encoder ${\mu }_{\varphi },{\sigma }_{\varphi }$, and the input $x$

So the encoder's output distribution is written

$q_{\varphi }(z\mid x)=\mathcal{N}({\mu }_{\varphi }(x),{\sigma }_{\varphi }^2(x)).$

Here $q_{\varphi }(z\mid x)$ reads "the distribution $q$, controlled by $\varphi$, over latent $z$ given the input $x$." The bar $\mid$ means "given / conditioned on". This $q$ is the encoder's guess; the details of why it takes this shape come from the ELBO derivation and live inside the full VAE.

8. The gradient symbol ${\nabla }_{\varphi }$ and the transform $g_{\varphi }$

Everything in section 2's promise pays off here: $g_{\varphi }$ is literally scale by $\sigma$, then shift by $\mu$ — the two knobs of a bell curve, applied by hand to a fixed noise sample.

9. Element-wise multiply $\odot$ and the identity matrix $I_d$

You need this only for the multivariate case; the one-dimensional intuition is identical, repeated $d$ times.

How these foundations feed the trick

The map below is not a restatement of headings — it shows the dependency of ideas and where the gradient breaks vs. flows: the left branch is the naive path that snaps at the random draw; the right branch is how the transform reroutes the same average so backprop survives.

Related tricks that solve the same gradient-through-randomness problem in other settings: 5.2.03-Gumbel-Softmax (for discrete samples), and 6.1.02-Policy-Gradients (an alternative estimator when reparameterization isn't available). The 4.3.01-KL-Divergence appears elsewhere in the VAE loss but is not needed to understand the trick itself.

Equipment checklist

Can you answer each without peeking? Reveal to check.