Foundations — Variational Autoencoders (VAE) theory

The parent note on VAE theory moves fast — it assumes you already read its vectors, distributions, averages, integrals, and divergences fluently. This page slows all the way down and earns each of those symbols from nothing. We introduce them one at a time, in an order where every symbol depends only on the ones already defined. Nothing below is used before it appears in its own boxed definition.

1. Data as a point in a huge space: $\mathbf{x}$

The picture. Imagine every image you could ever draw. A $28\times 28$ image has $784$ pixels, so it is a single point in a space with $784$ axes. You cannot draw $784$ axes, so we always pretend it is a 2D or 3D space — one dot = one whole image.

Why the topic needs it. VAEs are generative: we want to invent new points $\mathbf{x}$ that look like they came from the same cloud of real data. So we must first agree that a data example is a point in a space. The bold font just says "this is a vector, not a single number."

2. The hidden cause: latent variable $\mathbf{z}$

The picture. Squeeze the giant $\mathbf{x}$-cloud down onto a tiny 2D map. Nearby points on the map should decode to similar images. That tiny map is the latent space, and a point on it is $\mathbf{z}$.

Why the topic needs it. The whole latent variable model idea is: "data is complicated, but it was generated from a few simple hidden causes." Learning those causes gives us knobs to turn to generate new data. See Latent Variable Models for the general framework.

3. Uncertainty: probability distributions and $p(\cdot )$

Everything in a VAE is probabilistic — the model never says "the answer is exactly this," it says "here is a spread of plausible answers." So we need the language of distributions.

The picture. A hill. Tall where values are common, flat where they are rare. The whole hill has area $1$.

Why the topic needs it. The VAE's generative story is written entirely in these: sample $\mathbf{z}\sim p(\mathbf{z})$, then sample $\mathbf{x}\sim p(\mathbf{x}\mid \mathbf{z})$. The squiggle $\sim$ means "is drawn at random from."

4. The all-ones diagonal: identity matrix $\mathbf{I}$

Before the bell curve, we need one small piece of bookkeeping notation it will lean on.

The picture. For two dimensions, $\mathbf{I}=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right].$ Diagonal $=$ "how much each axis wiggles"; off-diagonal $=$ "how much axes wiggle together" — here, zero.

Why the topic needs it. VAEs assume the latent dimensions are independent — turning one knob does not drag another. That independence is exactly what the $0$s in $\mathbf{I}$ encode, which is why the prior is written $\mathcal{N}(0,\mathbf{I})$ once we meet the Gaussian next.

5. The bell curve: Gaussian $\mathcal{N}(\mathbf{\mu },{\mathbf{\sigma }}^2)$

Why this distribution and not another? Three reasons the parent note names:

1. Convenient math — sums and divergences of Gaussians have clean closed forms (no messy integrals to approximate).
2. Infinite support — every point has nonzero probability, so the latent map has no forbidden regions; you can decode anywhere.
3. Central Limit Theorem — pile up many small random effects and the total tends to a Gaussian, so it is a natural default.

6. The average machine: expectation $\mathbb{E}[\cdot ]$

The picture. Sample $\mathbf{z}$ a million times from the hill $p$, compute $f$ each time, and take the plain average. Common values of $\mathbf{z}$ pull the average toward their $f$.

Why the topic needs it. Every VAE loss term is an average of the form ${\mathbb{E}}_{q_{\varphi }(\mathbf{z}\mid \mathbf{x})}[\cdots ]$ — "average reconstruction quality over the codes the encoder suggests." Training estimates this average with just a few samples (Monte Carlo). Understanding $\mathbb{E}$ as "weighted average" is the key to reading every loss term you will meet later.

7. The integral $\int \cdots d\mathbf{z}$ — and why it is "intractable"

The picture. Slice the hill into thin vertical strips, measure each strip's area, add them all up.

Why "intractable" in the parent note. The marginal is $p_{\theta }(\mathbf{x})=\int p_{\theta }(\mathbf{x}\mid \mathbf{z})p(\mathbf{z})d\mathbf{z}.$ To compute it you must visit every $\mathbf{z}$ and ask "could this code have made $\mathbf{x}$?" In a $784$-slot or even $32$-slot latent space, there are effectively infinite $\mathbf{z}$ to check — no computer can sum them all. This impossibility is the entire reason VAEs exist: they dodge the integral with an approximation. That approximation is inference, formalised in the Evidence Lower Bound.

8. The logarithm $\log$ — why it appears everywhere

Why this tool. Likelihoods multiply across a dataset (${\prod }_{i}p({\mathbf{x}}^{(i)})$), giving astronomically small numbers. Taking $\log$ converts that product into a sum ${\sum }_i\log p({\mathbf{x}}^{(i)})$ that a computer can actually add without rounding to zero. It also linearises the Gaussian's exponential, exposing the clean quadratic terms you see in the KL derivation.

9. Comparing two distributions: KL divergence $D_{KL}$

The picture. Two bell curves. When they sit on top of each other, $D_{KL}=0$. Slide one away or squash its width and $D_{KL}$ climbs.

Why the topic needs it. The VAE's regularization term is $D_{KL}(q_{\varphi }(\mathbf{z}\mid \mathbf{x})\Vert p(\mathbf{z}))$ — it pulls the encoder's output toward the standard Gaussian prior, keeping the latent map smooth and gap-free. Full details in Kullback-Leibler Divergence.

10. Two networks and their parameters: $\theta$ and $\varphi$

The picture. An hourglass: data $\mathbf{x}$ funnels down through the encoder into the narrow waist $\mathbf{z}$, then fans back out through the decoder into a reconstruction $\hat{\mathbf{x}}$.

Why the topic needs it. Training adjusts $\theta$ and $\varphi$ together to maximize the Evidence Lower Bound. $\varphi$ is the inference machine (approximate posterior); $\theta$ is the generator.

Prerequisite map

Equipment checklist

Test yourself — reveal only after answering aloud.