Visual walkthrough — Variational Autoencoders (VAE) theory

This page rebuilds the single most important equation of the VAE — the Evidence Lower Bound — from absolutely nothing. No symbol appears before we draw it. If you have never seen an integral or a logarithm's tricks, start at line one and keep going.

---

Step 1 — What we actually want: the likelihood of the data

WHAT. Every generative model has a hidden dial we call $\mathbf{z}$ — the latent code. Think of $\mathbf{z}$ as a short list of numbers that means something ("a 7", "tilted left", "thick strokes"). The model turns $\mathbf{z}$ into a picture $\mathbf{x}$. We write:

$p_{\theta }(\mathbf{x})=\int p_{\theta }(\mathbf{x}\mid \mathbf{z})p(\mathbf{z})d\mathbf{z}$

Read every symbol:

• $\mathbf{x}$ — one real data point (an image), a fixed thing we were handed.
• $\mathbf{z}$ — the hidden code, a point we are free to choose.
• $p(\mathbf{z})$ — the prior: how likely each code is before seeing any data. We pick this to be a bell curve, $\mathcal{N}(0,\mathbf{I})$ (see Latent Variable Models).
• $p_{\theta }(\mathbf{x}\mid \mathbf{z})$ — the decoder: "given code $\mathbf{z}$, how likely is picture $\mathbf{x}$?" The subscript $\theta$ is the bag of neural-network weights we will train.
• $\int \dots d\mathbf{z}$ — the integral: "add up over every possible code."

WHY the integral. A picture could have come from many codes. To score how likely $\mathbf{x}$ is overall, we must weigh every code by how probable it is ($p(\mathbf{z})$) times how well it explains $\mathbf{x}$ ($p_{\theta }(\mathbf{x}\mid \mathbf{z})$), then sum. That summing-over-all-possibilities is exactly what $\int$ means.

PICTURE. The curve is $p(\mathbf{z})$. At each $\mathbf{z}$ a red bar shows the product $p_{\theta }(\mathbf{x}\mid \mathbf{z})p(\mathbf{z})$. The total shaded area under the red is $p_{\theta }(\mathbf{x})$.

---

Step 2 — Why we cannot just compute it

WHAT. $\mathbf{z}$ is not one number — it is a list of, say, 64 numbers. To do the integral by brute force we would chop each of the 64 axes into (say) 100 slices and add up $100^{64}$ boxes.

WHY it is hopeless. $100^{64}$ is more boxes than atoms in the universe. For all but toy problems the integral is intractable — we cannot evaluate it, and worse, we cannot differentiate it to train $\theta$.

PICTURE. One axis of $\mathbf{z}$ is easy to grid. Two axes: a grid of dots. The caption reminds you the real space has dozens of axes — the red dot is one lonely sample lost in an exponential ocean.

---

Step 3 — Smuggle the encoder in without changing anything

WHAT. Start from the log of what we want. We take the logarithm because it turns the product-and-integral into sums, which are far easier to handle:

$\log p_{\theta }(\mathbf{x})=\log \int p_{\theta }(\mathbf{x}\mid \mathbf{z})p(\mathbf{z})d\mathbf{z}$

Now multiply the inside by $\frac{q_{\varphi }(\mathbf{z}\mid \mathbf{x})}{q_{\varphi }(\mathbf{z}\mid \mathbf{x})}$, which equals $1$:

$\log p_{\theta }(\mathbf{x})=\log \int q_{\varphi }(\mathbf{z}\mid \mathbf{x})\frac{p_{\theta }(\mathbf{x}\mid \mathbf{z})p(\mathbf{z})}{q_{\varphi }(\mathbf{z}\mid \mathbf{x})}d\mathbf{z}$

Term by term:

• $q_{\varphi }(\mathbf{z}\mid \mathbf{x})$ — the encoder / guide: a bell curve over codes, centred on the good region for this particular $\mathbf{x}$. $\varphi$ is its own set of weights.
• Multiplying by $\frac{q}{q}=1$ changes nothing mathematically — but it lets us read the integral as an average with respect to $q$.

WHY. "$\int q(\cdot )f(\cdot )d\mathbf{z}$" is precisely the definition of an average of $f$ where each code is weighted by how often the guide $q$ would pick it. We write that average as ${\mathbb{E}}_{q_{\varphi }}[\cdot ]$:

$\log p_{\theta }(\mathbf{x})=\log {\mathbb{E}}_{q_{\varphi }(\mathbf{z}\mid \mathbf{x})}\left[\frac{p_{\theta }(\mathbf{x}\mid \mathbf{z})p(\mathbf{z})}{q_{\varphi }(\mathbf{z}\mid \mathbf{x})}\right]$

The bracket is called the ratio $R=\frac{p_{\theta }(\mathbf{x}\mid \mathbf{z})p(\mathbf{z})}{q_{\varphi }(\mathbf{z}\mid \mathbf{x})}$. We will use $R$ in the next step.

PICTURE. Two curves: the wide prior $p(\mathbf{z})$ in black, the narrow guide $q_{\varphi }(\mathbf{z}\mid \mathbf{x})$ in red, hugging the region that explains $\mathbf{x}$. Instead of sampling everywhere, we will now sample only under the red hump.

---

Step 4 — The one inequality: Jensen

WHAT. We have $\log$ of an average, $\log {\mathbb{E}}_{q_{\varphi }}[R]$. We want to move the $\log$ inside, giving ${\mathbb{E}}_{q_{\varphi }}[\log R]$. These are not equal — but there is a guaranteed direction:

$\log {\mathbb{E}}_{q_{\varphi }}[R]\ge {\mathbb{E}}_{q_{\varphi }}[\log R]$

(The subscript $q_{\varphi }$ is essential: both averages are taken with respect to the encoder's distribution $q_{\varphi }(\mathbf{z}\mid \mathbf{x})$ — that is the whole reason we smuggled it in.)

WHY log, and why this direction. The graph of $\log$ is concave — it bends downward like a dome. For any dome-shaped function, the height above the average input is at least the average of the heights. That is Jensen's inequality. We use $\log$ specifically because (a) it is concave so Jensen applies, and (b) it splits products into sums (Step 5). Applying it:

$\log p_{\theta }(\mathbf{x})\ge {\mathbb{E}}_{q_{\varphi }(\mathbf{z}\mid \mathbf{x})}\left[\log \frac{p_{\theta }(\mathbf{x}\mid \mathbf{z})p(\mathbf{z})}{q_{\varphi }(\mathbf{z}\mid \mathbf{x})}\right]$

The right-hand side is a floor under $\log p_{\theta }(\mathbf{x})$. We name it $\mathcal{L}$ — the Evidence Lower Bound, or ELBO.

PICTURE. The red dome is $\log$. Pick two points on the curve; the straight red chord between them lies below the curve. The chord's midpoint height = ${\mathbb{E}}_{q_{\varphi }}[\log R]$; the curve's height above the mean input = $\log {\mathbb{E}}_{q_{\varphi }}[R]$. The gap between them is the slack in Jensen.

---

Step 5 — Split the log into two meaningful jobs

WHAT. Use $\log \frac{ab}{c}=\log a+\log b-\log c$ to break the ELBO apart:

$\mathcal{L}={\mathbb{E}}_{q_{\varphi }}[\log p_{\theta }(\mathbf{x}\mid \mathbf{z})]+{\mathbb{E}}_{q_{\varphi }}\left[\log \frac{p(\mathbf{z})}{q_{\varphi }(\mathbf{z}\mid \mathbf{x})}\right]$

The second average is, by definition, the negative KL divergence between guide and prior:

${\mathbb{E}}_{q_{\varphi }}\left[\log \frac{p(\mathbf{z})}{q_{\varphi }(\mathbf{z}\mid \mathbf{x})}\right]=-D_{KL}(q_{\varphi }(\mathbf{z}\mid \mathbf{x})\Vert p(\mathbf{z}))$

Term by term:

• Reconstruction — sample a code from the guide, decode it, ask "did we get $\mathbf{x}$ back?" High when the decoder is faithful.
• Regulariser — $D_{KL}(\cdot \Vert \cdot )\ge 0$ measures how far the guide's bell curve has drifted from the standard prior. Subtracting it punishes drift, keeping the latent space smooth and samplable (see Evidence Lower Bound (ELBO)).

WHY this split matters. These two terms pull against each other. Perfect reconstruction wants sharp, spread-out codes; the KL wants every code squashed toward the origin. VAE training is the tug-of-war between them. (When the KL wins too hard, the code is ignored — that failure is Posterior Collapse.)

PICTURE. A balance beam: left pan "reconstruction ↑", right pan "KL to prior ↓", the red pivot is $\mathcal{L}$ we maximise.

---

Step 6 — Exactly how big is the gap? (the degenerate/tight case)

WHAT. Jensen gave "$\ge$" but hid by how much. We now name the gap precisely. Claim:

$\log p_{\theta }(\mathbf{x})=\mathcal{L}(\theta ,\varphi ;\mathbf{x})+D_{KL}(q_{\varphi }(\mathbf{z}\mid \mathbf{x})\Vert p_{\theta }(\mathbf{z}\mid \mathbf{x}))$

Derivation. Start from the KL between the guide and the true posterior $p_{\theta }(\mathbf{z}\mid \mathbf{x})$, and use Bayes' rule $p_{\theta }(\mathbf{z}\mid \mathbf{x})=\frac{p_{\theta }(\mathbf{x}\mid \mathbf{z})p(\mathbf{z})}{p_{\theta }(\mathbf{x})}$:

$D_{KL}(q_{\varphi }\Vert p_{\theta }(\mathbf{z}\mid \mathbf{x}))={\mathbb{E}}_{q_{\varphi }}[\log q_{\varphi }-\log p_{\theta }(\mathbf{x}\mid \mathbf{z})-\log p(\mathbf{z})+\log p_{\theta }(\mathbf{x})]$

Because $\log p_{\theta }(\mathbf{x})$ has no $\mathbf{z}$ in it, its average is itself, and the remaining three terms are exactly $-\mathcal{L}$. So:

$D_{KL}(q_{\varphi }\Vert p_{\theta }(\mathbf{z}\mid \mathbf{x}))=\log p_{\theta }(\mathbf{x})-\mathcal{L}$

WHY this closes the loop. KL is never negative. So $\log p_{\theta }(\mathbf{x})-\mathcal{L}\ge 0$ — a second, airtight proof that $\mathcal{L}$ is a lower bound. And the gap vanishes exactly when the guide equals the true posterior, $q_{\varphi }=p_{\theta }(\mathbf{z}\mid \mathbf{x})$: the degenerate case where the bound is tight and we lose nothing.

PICTURE. A vertical stack: top level = $\log p_{\theta }(\mathbf{x})$ (the ceiling), bottom level = $\mathcal{L}$ (our floor), and the red bracket between them labelled $D_{KL}(q_{\varphi }\Vert p_{\theta }(\mathbf{z}\mid \mathbf{x}))\ge 0$. When the red bracket shrinks to zero, floor touches ceiling.

---

Step 7 — The KL term has a clean closed form (so we can actually train)

WHAT. For a diagonal-Gaussian guide $q=\mathcal{N}(\mathbf{\mu },{\mathbf{\sigma }}^2\mathbf{I})$ against prior $p=\mathcal{N}(0,\mathbf{I})$ in $d$ dimensions, the KL is a plain formula — no integral left:

$D_{KL}(q\Vert p)=\frac{1}{2}{\sum }_{j=1}^d({\mu }_j^2+{\sigma }_j^2-1-\log {\sigma }_j^2)$

Term by term for a single dimension:

• ${\mu }_j^2$ — punishes the mean for straying from $0$.
• ${\sigma }_j^2$ — punishes the spread for being wider than the prior's $1$.
• $-1$ — the baseline so a perfect match costs $0$.
• $-\log {\sigma }_j^2$ — punishes the spread for collapsing to $0$ (too sharp).

WHY it matters. Because this is elementary arithmetic on the encoder's outputs ${\mu }_j,{\sigma }_j$, we can differentiate it and let gradients flow. Paired with the Reparameterization Trick for the reconstruction term, the whole ELBO becomes trainable by ordinary backprop.

Sanity / degenerate check. Set ${\mu }_j=0,{\sigma }_j=1$ (guide equals prior): each summand is $0+1-1-\log 1=0$. Zero cost, as it must be.

PICTURE. The KL cost of one dimension as a red bowl over $(\mu ,\sigma )$; the bottom of the bowl sits exactly at $\mu =0,\sigma =1$ with value $0$, and the walls rise as you drift or collapse.

---

The one-picture summary

The whole story on one canvas: the impossible ceiling $\log p_{\theta }(\mathbf{x})$, the guide $q_{\varphi }$ we hired, Jensen giving us a floor $\mathcal{L}$, the floor splitting into reconstruction minus KL, and the red gap that shrinks to zero when the guide matches the true posterior.

Recall Feynman retelling — say it like you mean it

We wanted the probability the model gives to a real picture. To get it you'd have to check every possible hidden code — there are astronomically many, so it's impossible. Trick: hire an encoder that, for each picture, points straight at the few codes that matter. Take a logarithm (turns messy products into sums and, being dome-shaped, lets us use Jensen's inequality). Jensen hands us a floor underneath the impossible number — the ELBO. Break the ELBO into two honest jobs: "rebuild the picture from its code" minus "keep the code's distribution close to a plain bell curve." Those two fight each other, and training balances them. Finally we proved the exact size of the gap between the impossible ceiling and our floor: it's the KL from our encoder to the true best guide, and it's zero when they match. Since KL can't be negative, our floor is always a real floor — so pushing it up drags the true likelihood up with it. And because the KL-to-the-prior part has a tiny closed-form formula, the whole thing trains with plain calculus.

Recall

What is intractable in $p_{\theta }(\mathbf{x})=\int p_{\theta }(\mathbf{x}\mid \mathbf{z})p(\mathbf{z})d\mathbf{z}$? ::: The integral over all high-dimensional codes $\mathbf{z}$ — exponentially many to sum. Which property of $\log$ powers Jensen's inequality here? ::: It is concave (dome-shaped), so $\log {\mathbb{E}}_{q_{\varphi }}[R]\ge {\mathbb{E}}_{q_{\varphi }}[\log R]$. Name the two terms of the ELBO. ::: Reconstruction (expected $\log p_{\theta }(\mathbf{x}\mid \mathbf{z})$) minus KL divergence of the encoder from the prior. Exactly how big is the gap between $\log p_{\theta }(\mathbf{x})$ and the ELBO? ::: $D_{KL}(q_{\varphi }(\mathbf{z}\mid \mathbf{x})\Vert p_{\theta }(\mathbf{z}\mid \mathbf{x}))$, which is $\ge 0$ and zero when the guide equals the true posterior. What is the closed-form KL for $\mathcal{N}(\mu ,{\sigma }^2)$ vs $\mathcal{N}(0,1)$ per dimension? ::: $\frac{1}{2}({\mu }^2+{\sigma }^2-1-\log {\sigma }^2)$.

---

Related: Reparameterization Trick · Kullback-Leibler Divergence · Evidence Lower Bound (ELBO) · Posterior Collapse · Conditional VAE · Normalizing Flows · Generative Adversarial Networks (GANs) · Disentangled Representations