Visual walkthrough — ELBO objective and KL term

This page rebuilds the ELBO one drawing at a time. We start with nothing but the idea of a probability and end holding the full training loss of a VAE. Every symbol is earned before it is used. If you have never seen an integral or the letter $\mathbb{E}$, start at line one — you will be fine.

---

Step 0 — The characters and their pictures

Before any equation, meet the cast. A distribution is just a shape that says "how likely is each value?" — a hill over a number line, where taller = more likely, and the total area under the hill is exactly $1$ (something must happen).

The whole game: we want to make the data likely, but the true code-belief $p_{\theta }(z\mid x)$ is out of reach, so we hire a stand-in $q_{\varphi }(z\mid x)$.

---

Step 1 — The ceiling we cannot reach

WHAT. We want to push up $\log p_{\theta }(x)$, the log-probability the model assigns to the real data. Why the $\log$? Because probabilities of a whole dataset multiply, and $\log$ turns products into sums — sums are what optimizers like.

WHY it's hard. The model only defines $x$ through a code: to get $p_{\theta }(x)$ alone we must add up every possible code, $p_{\theta }(x)=\underset{\text{sum over}}{\underbrace{\int }}\underset{\text{decoder}}{\underbrace{p_{\theta }(x\mid z)}}\underset{\text{prior}}{\underbrace{p(z)}}dz.$ The $\int \dots dz$ is "sum over all codes $z$" — a continuous sum. For a neural decoder there is no formula for this sum, and there are infinitely many codes. So $\log p_{\theta }(x)$ is a real number we can name but never compute.

PICTURE. A ceiling drawn as a dashed line high above. We can point at it but not touch it — so we will build a tower underneath and read the tower's height instead.

---

Step 2 — Smuggle in the encoder (multiply by 1)

WHAT. Inside the sum we multiply by $\frac{q_{\varphi }(z\mid x)}{q_{\varphi }(z\mid x)}$ — which equals $1$, so nothing changes: $\log p_{\theta }(x)=\log \int q_{\varphi }(z\mid x)\cdot \frac{p_{\theta }(x,z)}{q_{\varphi }(z\mid x)}dz.$ Here $p_{\theta }(x,z)=p_{\theta }(x\mid z)p(z)$ is the joint — the decoder and prior glued together.

WHY. A sum of "(shape $q$) times (something)" is exactly the definition of an average over $q$. So the right side is $\log p_{\theta }(x)=\log {\mathbb{E}}_{q_{\varphi }(z\mid x)}\left[\frac{p_{\theta }(x,z)}{q_{\varphi }(z\mid x)}\right].$ We chose $q_{\varphi }$ because it is a shape we built and can sample from — that is what makes the average estimable later. This is the heart of variational inference.

PICTURE. The ratio $\frac{p_{\theta }(x,z)}{q_{\varphi }(z\mid x)}$ is a bumpy curve; sprinkling sample codes from $q_{\varphi }$ and reading the curve at each dot approximates the average — we could never do that with the raw integral of Step 1.

---

Step 3 — Jensen's inequality: slide the log inside

WHAT. We move $\log$ from outside the average to inside: $\log {\mathbb{E}}_{q_{\varphi }(z\mid x)}\left[R(z)\right]\ge {\mathbb{E}}_{q_{\varphi }(z\mid x)}\left[\log R(z)\right],R(z)=\frac{p_{\theta }(x,z)}{q_{\varphi }(z\mid x)}.$

WHY this tool and not another. $\log$ is concave — its graph bulges upward like a dome, every chord lies below the curve. For any concave curve, "the curve read at the average input" sits above "the average of the curve's outputs". That is Jensen's inequality, and it is the one tool that lets a $\log$ jump inside an $\mathbb{E}$ while promising the result only ever gets smaller. Smaller is exactly what we want: a guaranteed lower bound.

PICTURE. Two points on the log-dome; the straight chord between them is the "average of the logs", the dome above the chord's midpoint is the "log of the average". The dome is higher — so $\log \mathbb{E}\ge \mathbb{E}\log$, the gap is the bulge.

The right side is our tower — the ELBO: $\mathcal{L}(\theta ,\varphi ;x)={\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p_{\theta }(x,z)-\log q_{\varphi }(z\mid x)].$

---

Step 4 — Split the tower into Reconstruction and KL

WHAT. Expand $\log p_{\theta }(x,z)=\log p_{\theta }(x\mid z)+\log p(z)$ and regroup: $\mathcal{L}=\underset{\text{(A) reconstruction}}{\underbrace{{\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p_{\theta }(x\mid z)]}}-\underset{\text{(B) }{D}_{KL}(q_{\varphi }\Vert p)}{\underbrace{{\mathbb{E}}_{q_{\varphi }(z\mid x)}\left[\log \frac{q_{\varphi }(z\mid x)}{p(z)}\right]}}.$

WHY. Two jobs fall out cleanly:

• (A) rewards codes that let the decoder rebuild $x$ — high when the reconstruction is faithful.
• (B) is the KL divergence: the average extra surprise of using $q_{\varphi }$'s codes while pretending they came from the prior $p$. It pulls $q_{\varphi }$ back toward the bell curve, the regularizer.

PICTURE. A tug-of-war rope: reconstruction pulls $q_{\varphi }$ toward wherever the data-specific code lives; KL pulls it back toward the prior at $0$. Their balance point is the trained encoder.

---

Step 5 — What the gap actually is

WHAT. How far below the ceiling does the tower stop? We track it in four small, honest moves. Start from Bayes' rule (the definition of the true posterior $p_{\theta }(z\mid x)$ from Step 0), take logs: $\log p_{\theta }(x)=\log p_{\theta }(x\mid z)+\log p(z)-\log p_{\theta }(z\mid x).$

Move 1 — average both sides over the encoder. The left side has no $z$ in it, so averaging leaves it untouched; the right side becomes three expectations: $\log p_{\theta }(x)={\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p_{\theta }(x\mid z)]+{\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p(z)]-{\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p_{\theta }(z\mid x)].$ Why average? Because our whole objective already lives as an average over $q_{\varphi }$; putting the identity in the same currency lets us match terms.

Move 2 — add and subtract ${\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log q_{\varphi }(z\mid x)]$. This adds zero, so the value is unchanged — but it plants the $\log q_{\varphi }$ we need to build two KL divergences (each KL is a difference of two logs averaged over $q_{\varphi }$). $\log p_{\theta }(x)={\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p_{\theta }(x\mid z)]+{\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p(z)]-{\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log q_{\varphi }(z\mid x)]+{\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log q_{\varphi }(z\mid x)]-{\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p_{\theta }(z\mid x)].$

Move 3 — group the two KL-shaped pieces. The middle three terms are exactly ${\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p(z)-\log q_{\varphi }(z\mid x)]=-D_{KL}(q_{\varphi }\Vert p)$, and the last two are ${\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log q_{\varphi }(z\mid x)-\log p_{\theta }(z\mid x)]=+D_{KL}(q_{\varphi }\Vert p_{\theta }(z\mid x))$: $\log p_{\theta }(x)=\underset{\mathcal{L}(\theta ,\varphi ;x)}{\underbrace{{\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p_{\theta }(x\mid z)]-D_{KL}(q_{\varphi }\Vert p)}}+\underset{\text{the gap}}{\underbrace{D_{KL}(q_{\varphi }(z\mid x)\Vert p_{\theta }(z\mid x))}}.$

Move 4 — read it. $\log p_{\theta }(x)=\mathcal{L}(\theta ,\varphi ;x)+D_{KL}(q_{\varphi }(z\mid x)\Vert p_{\theta }(z\mid x)).$

WHY it matters. The missing height is exactly the KL between our stand-in $q_{\varphi }$ and the true posterior $p_{\theta }(z\mid x)$. Because KL is never negative, the tower is never above the ceiling — the bound is honest. And it names our two levers precisely.

PICTURE. Ceiling = $\log p_{\theta }(x)$; tower top = $\mathcal{L}$; the shaded strip between them = the gap-KL. Squeeze that strip to zero and tower meets ceiling.

---

Step 6 — Making the terms concrete: Gaussian encoder and decoder

WHAT. To turn everything into numbers we must pick actual shapes for all three players:

• Prior: the plain bell curve $p(z)=\mathcal{N}(0,I)$.
• Encoder: a bell curve shifted and stretched per data point, $q_{\varphi }(z\mid x)=\mathcal{N}(\mu ,\mathrm{diag}({\sigma }^2))$. The network reads $x$ and outputs two lists: $\mu =({\mu }_1,\dots ,{\mu }_d)$ (where each code-axis is centred) and $\sigma =({\sigma }_1,\dots ,{\sigma }_d)$ (how wide/uncertain each axis is). "$\mathrm{diag}$" means axes are independent, so the KL is a sum over axes.
• Decoder: a bell curve over the $D$-dimensional data, centred at the network output ${\mu }_{\theta }(z)$ with unit variance, $p_{\theta }(x\mid z)=\mathcal{N}(x;{\mu }_{\theta }(z),I)$.

WHY the decoder shape matters (this fixes the reconstruction term). A $D$-dimensional unit-variance Gaussian has log-density $\log p_{\theta }(x\mid z)=\log \mathcal{N}(x;{\mu }_{\theta }(z),I)=-\frac{1}{2}\Vert x-{\mu }_{\theta }(z){\Vert }^2-\frac{D}{2}\log (2\pi ).$ The first piece is negative squared reconstruction error (rewards a faithful rebuild); the second piece $-\frac{D}{2}\log (2\pi )$ is a constant — it does not depend on $\theta$ or $\varphi$, so it never changes the gradient, which is why it is often dropped in code. But it is part of the true log-likelihood, so when we report an actual ELBO number we must keep it.

WHY the Gaussian KL has a closed form. With two bell curves the KL integral solves exactly — no sampling. Per axis $j$, starting from $D_{KL}={\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log q_{\varphi }(z_j)-\log p(z_j)]$ and using $\mathbb{E}[z_j^2]={\mu }_j^2+{\sigma }_j^2$ and $\mathbb{E}[(z_j-{\mu }_j{)}^2]={\sigma }_j^2$: $D_{KL}(q_{\varphi }\Vert p)=\frac{1}{2}{\sum }_{j=1}^d(\underset{\text{off-centre penalty}}{\underbrace{{\mu }_j^2}}+\underset{\text{too wide}}{\underbrace{{\sigma }_j^2}}-\underset{\text{too narrow}}{\underbrace{\log {\sigma }_j^2}}-1).$ Read the terms: ${\mu }_j^2$ punishes drifting from the prior's centre $0$; ${\sigma }_j^2$ punishes being wider than the prior; $-\log {\sigma }_j^2$ explodes as ${\sigma }_j\to 0$, forbidding a razor-thin encoder; the $-1$ makes the whole thing exactly $0$ when ${\mu }_j=0,{\sigma }_j=1$ (encoder equals prior).

PICTURE. Each axis draws two bells: fixed prior (centre $0$, width $1$) versus encoder (${\mu }_j$, ${\sigma }_j$). The KL is the total mismatch; it bottoms out at zero only when the two bells coincide.

Now the edge cases the reader might hit:

• Perfect encoder ($q_{\varphi }=p_{\theta }(z\mid x)$): gap-KL $=0$, tower touches ceiling, $\mathcal{L}=\log p_{\theta }(x)$. The dream case.
• No KL term at all (drop (B)): reconstruction pushes each $x$ to its own private, spiky code; the latent space becomes discontinuous — useless for sampling.
• KL dominates (${\mu }_j\to 0,{\sigma }_j\to 1$ for every $x$): KL $=0$ but codes carry no information about $x$ — the decoder ignores $z$, another face of posterior collapse. This is the information-bottleneck pinched shut.
• Loose bound (big gap-KL): $\mathcal{L}$ still valid but far below $\log p_{\theta }(x)$; tighten with richer $q_{\varphi }$ — e.g. flows or importance-weighted bounds.
• ${\sigma }_j\to 0$ (encoder certain): $-\log {\sigma }_j^2\to +\infty$, KL explodes — the objective forbids a zero-variance encoder, a built-in safety rail.

---

Worked numbers (matching the parent)

---

The one-picture summary

Recall Feynman retelling — say it back in plain words

We wanted to make the data likely, but the honest score $\log p(x)$ needs an impossible sum over every hidden code. So we hired a stand-in encoder $q_{\varphi }$ we can sample from, snuck it into the sum as a disguised $1$, and used Jensen's rule (log-dome sits above its chords) to slide the log inside — turning the impossible score into a computable tower, the ELBO, guaranteed to sit below the true ceiling. The tower splits into two forces: reconstruction pulling codes toward whatever rebuilds $x$ (a Gaussian decoder makes this negative squared error plus a fixed constant), and KL pulling them back toward the bell-curve prior. When both encoder and prior are Gaussians the KL becomes a tidy sum $\frac{1}{2}\sum ({\mu }_j^2+{\sigma }_j^2-\log {\sigma }_j^2-1)$. The height still missing from the ceiling is exactly how wrong our stand-in is — the KL to the true posterior $p_{\theta }(z\mid x)$. Close that gap and the tower kisses the ceiling. Kill the KL and codes go feral; let KL win and codes forget the data; drive $\sigma \to 0$ and the KL screams infinity — so the math itself keeps us in the sweet middle.

Recall Quick checks

Which inequality lets log jump inside the average, and why does it point down? ::: Jensen's; $\log$ is concave so $\log \mathbb{E}[R]\ge \mathbb{E}[\log R]$ — a lower bound. The gap between ELBO and $\log p_{\theta }(x)$ equals what? ::: $D_{KL}(q_{\varphi }(z\mid x)\Vert p_{\theta }(z\mid x))$. Which force stops each $x$ from grabbing a private spiky code? ::: The KL term pulling $q_{\varphi }$ toward the prior $p(z)$. In the Gaussian KL formula, which term blows up as ${\sigma }_j\to 0$? ::: $-\log {\sigma }_j^2\to +\infty$. What shape did we assume for the decoder, and what does its log-density become? ::: $\mathcal{N}(x;{\mu }_{\theta }(z),I)$; its log-density is $-\frac{1}{2}\Vert x-{\mu }_{\theta }(z){\Vert }^2-\frac{D}{2}\log (2\pi )$.