Worked examples — ELBO objective and KL term

This page is a workshop. The parent note built the theory: the ELBO is a lower bound on $\log p_{\theta }(x)$, it splits into a reconstruction reward minus a KL penalty, and the gap to the true log-likelihood is itself a KL. Here we use those results on every kind of input the topic can hand you — normal cases, zero cases, degenerate cases, limits, a real word problem, and an exam twist.

Recall Symbols we will reuse (from the parent)

• $x$ ::: the observed data point (e.g. a small vector)
• $z$ ::: the hidden "latent code" a VAE invents to explain $x$
• $q_{\varphi }(z\mid x)$ ::: the encoder — our guessed distribution over $z$ given $x$
• $p(z)=\mathcal{N}(0,I)$ ::: the prior — the "default" shape we want $z$ to follow
• $p_{\theta }(x\mid z)$ ::: the decoder — rebuilds $x$ from $z$
• $\mathcal{L}={\mathbb{E}}_q[\log p_{\theta }(x\mid z)]-D_{KL}(q_{\varphi }(z\mid x)\Vert p(z))$ ::: the ELBO
• $\mathcal{N}(\mu ,{\sigma }^2)$ ::: a bell curve (Gaussian) centred at $\mu$ with spread $\sigma$

Everything below leans on one workhorse formula from the parent, so let us re-anchor it before spending it.

---

The scenario matrix

Every ELBO exercise you will meet is one of these cells. The examples afterward tick each cell.

#	Cell class	What makes it special	Covered by
A	Perfect match $q=p$	$\mu =0,\sigma =1$ → KL should be $0$	Ex 1
B	Off-centre, right width	$\mu \ne 0,\sigma =1$ → only ${\mu }^2$ cost	Ex 2
C	Centred, wrong width	$\mu =0,\sigma \ne 1$ → only spread cost	Ex 3
D	Degenerate collapse	$\sigma \to 0$ → KL $\to \infty$ (limit)	Ex 4
E	Full single-point ELBO	reconstruction and KL together	Ex 5
F	Reverse-vs-forward KL asymmetry	$D_{KL}(q\Vert p)\ne D_{KL}(p\Vert q)$	Ex 6
G	Real-world word problem	choosing $\beta$, bits interpretation	Ex 7
H	Exam twist: the gap	recover $\log p_{\theta }(x)$ from ELBO + posterior KL	Ex 8

---

Example 1 — Cell A: the perfect match

Forecast: guess the number before reading on. If $q$ is the prior, how many "extra bits" do you pay? Zero — write that down.

1. Plug into the closed form. Each dimension contributes $\frac{1}{2}({\mu }_j^2+{\sigma }_j^2-\log {\sigma }_j^2-1)$. Why this step? The formula is exact for diagonal Gaussians; no sampling needed.
2. Evaluate one dimension. ${\mu }_j=0,{\sigma }_j=1$ gives $\frac{1}{2}(0+1-\log 1-1)=\frac{1}{2}(0+1-0-1)=0$. Why this step? $\log 1=0$, so all four pieces cancel — this is the "$-1$ was calibrated for exactly this case" moment.
3. Sum over $d=2$. $0+0=0$.

Verify: the parent's property "$D_{KL}=0$ iff $q=p$ a.e." demands exactly this. $D_{KL}=0.✓$

---

Example 2 — Cell B: off-centre, correct width

Forecast: width is perfect, so the $\sigma$-and-$\log$ pieces should self-cancel like in Ex 1. Only the ${\mu }^2$ terms survive. Guess: about $0.17$.

1. Kill the width terms. With ${\sigma }_j=1$: ${\sigma }_j^2-\log {\sigma }_j^2-1=1-0-1=0$. Why this step? Isolates the pure "off-centre" penalty so we see it cleanly.
2. Keep the mean terms. $D_{KL}=\frac{1}{2}({\mu }_1^2+{\mu }_2^2)=\frac{1}{2}(0.25+0.09)=\frac{1}{2}(0.34)$. Why this step? ${\mu }^2$ is the only surviving cost — it grows quadratically with drift, so being twice as far off costs four times as much.
3. Compute. $=0.17$.

Verify: must be $>0$ since $q\ne p$; and it equals half the squared distance of the mean from origin, $\frac{1}{2}\Vert \mu {\Vert }^2=\frac{1}{2}(0.5831{)}^2=\mathrm{0.17.}✓$

---

Example 3 — Cell C: centred but wrong width

Forecast: which is punished more — twice too wide, or twice too narrow? Guess before computing. (Hint: look at the $-\log {\sigma }^2$ term.)

1. Too wide, one dim. $\frac{1}{2}(0+4-\log 4-1)=\frac{1}{2}(4-1.386-1)=\frac{1}{2}(1.614)=0.807$. Why this step? ${\sigma }^2=4$ dominates; the $-\log {\sigma }^2$ gives back only a little.
2. Two dims wide. $2\times 0.807=1.614$.
3. Too narrow, one dim. ${\sigma }^2=0.25$: $\frac{1}{2}(0+0.25-\log 0.25-1)=\frac{1}{2}(0.25+1.386-1)=\frac{1}{2}(0.636)=0.318$. Why this step? Now ${\sigma }^2$ is tiny but $-\log {\sigma }^2=+1.386$ is large and positive — the narrowness penalty takes over.
4. Two dims narrow. $2\times 0.318=0.636$.

Verify: the two costs differ ($1.614\ne 0.636$) — the KL is not symmetric in "wide vs narrow". Look at the amber curve in the figure: it is steeper on the narrow ($\sigma <1$) side asymptotically but rises faster on the wide side for moderate deviations. Both $>0.✓$

---

Example 4 — Cell D: degenerate collapse ($\sigma \to 0$)

Forecast: a deterministic encoder pins $z$ to a single point. Does the model like this? Guess whether KL stays finite.

1. Isolate the dangerous term. Per dimension the KL contains $-\log {\sigma }_j^2=-2\log {\sigma }_j$. Why this step? As ${\sigma }_j\to 0^{+}$, $\log {\sigma }_j\to -\infty$, so $-2\log {\sigma }_j\to +\infty$.
2. Take the limit. ${\lim }_{\sigma \to 0^{+}}\frac{1}{2}({\mu }^2+{\sigma }^2-\log {\sigma }^2-1)=+\infty$. Why this step? The finite pieces ${\mu }^2$ and $-1$ are swamped; the $-\log {\sigma }^2$ term dominates.
3. Interpret. The KL penalty forbids infinite sharpness. This is the mathematical guardrail against one failure mode — but note it does not prevent Posterior Collapse, which is the opposite pathology ($q\to p$ so the code carries no info).

Verify (numerically): at $\sigma =0.1$, $-\log (0.01)=4.605$, KL-per-dim $\approx \frac{1}{2}(0.01+4.605-1)=1.808$; at $\sigma =0.01$, $-\log (10^{-4})=9.21$, KL-per-dim $\approx \frac{1}{2}(0.0001+9.21-1)=4.105$ — growing without bound. $✓$

---

Example 5 — Cell E: full single-point ELBO

Forecast: reconstruction is near-perfect (decoded point sits almost on $x$), so that term is $\approx 0$. The KL is a genuine positive cost. Guess: ELBO $\approx -0.9$.

1. Reconstruction term. For a unit-variance Gaussian decoder, $\log p_{\theta }(x\mid z)=-\frac{1}{2}\Vert x-{\mu }_{\theta }(z){\Vert }^2$ (plus a constant we drop). $\Vert x-{\mu }_{\theta }(z){\Vert }^2=(0.2-0.19{)}^2+(0.8-0.79{)}^2=0.0001+0.0001=0.0002$, so term $=-0.0001$. Why this step? The decoder's log-density is negative squared error for identity covariance — good reconstruction $\Rightarrow$ near-zero cost.
2. KL term, dim 1. $\mu =0.5,{\sigma }^2=0.64$: $0.25+0.64-\log 0.64-1=-0.11+0.446=0.336$. Why this step? $-\log 0.64=+0.446$ because $\sigma <1$ (slight narrowness cost).
3. KL term, dim 2. $\mu =-0.3,{\sigma }^2=1.44$: $0.09+1.44-\log 1.44-1=0.53-0.365=0.165$. Why this step? Here $\sigma >1$, so $-\log 1.44=-0.365$ refunds some cost.
4. Assemble KL. $D_{KL}=\frac{1}{2}(0.336+0.165)=\frac{1}{2}(0.501)=0.2503$.
5. ELBO. $\mathcal{L}=-0.0001-0.2503=-0.2504$.

Verify: ELBO must be $\le \log p_{\theta }(x)$. It is a small negative number dominated by the KL, as forecast (sign right, magnitude reasonable). Recompute KL directly: $\frac{1}{2}[(0.25+0.64-\log 0.64-1)+(0.09+1.44-\log 1.44-1)]=\mathrm{0.2503.}✓$

---

Example 6 — Cell F: reverse vs forward KL

Forecast: the parent said this is asymmetric. Guess whether swapping the arguments changes the number — yes — and which is bigger.

Use the general two-Gaussian formula (derive once, reuse): $D_{KL}(\mathcal{N}({\mu }_0,{\sigma }_0^2)\Vert \mathcal{N}({\mu }_1,{\sigma }_1^2))=\log \frac{{\sigma }_1}{{\sigma }_0}+\frac{{\sigma }_0^2+({\mu }_0-{\mu }_1{)}^2}{2{\sigma }_1^2}-\frac{1}{2}.$

1. Forward $D_{KL}(q\Vert p)$. ${\mu }_0=0,{\sigma }_0=1,{\mu }_1=1,{\sigma }_1=2$: $\log 2+\frac{1+1}{8}-\frac{1}{2}=0.6931+0.25-0.5=0.4431$. Why this step? This is the direction the ELBO actually optimises (expectation under the encoder $q$).
2. Reverse $D_{KL}(p\Vert q)$. Swap: ${\mu }_0=1,{\sigma }_0=2,{\mu }_1=0,{\sigma }_1=1$: $\log \frac{1}{2}+\frac{4+1}{2}-\frac{1}{2}=-0.6931+2.5-0.5=1.3069$. Why this step? Same two distributions, opposite roles — shows the numbers genuinely differ.

Verify: $0.4431\ne 1.3069$, confirming asymmetry. The reverse KL here is larger, which is why Variational Inference picks the mode-seeking forward-KL $D_{KL}(q\Vert p)$: expectations are taken under $q$ (which we can sample), giving the "zero-forcing" behaviour the parent described. See also Importance Weighted Autoencoders and Normalizing Flows for tighter alternatives. $✓$

---

Example 7 — Cell G: real-world word problem

Forecast: more $\beta$ = more pressure toward the prior = fewer bits used. Guess whether the objective goes up or down.

1. State the objective. $\beta$-VAE maximises ${\mathbb{E}}_q[\log p_{\theta }(x\mid z)]-\beta D_{KL}(q\Vert p)$. Why this step? The scalar $\beta$ re-weights the information-bottleneck trade-off between fidelity and compression.
2. Nats → bits. $1$ nat $=1/\ln 2=1.4427$ bits, so $12$ nats $=12\times 1.4427=17.31$ bits. Why this step? KL measures "extra bits/nats" — the parent's coding interpretation. Reporting in bits makes the channel capacity legible.
3. Weighted objective. ${\mathbb{E}}_q[\log p_{\theta }]-\beta D_{KL}=-40-3(12)=-40-36=-76$. Why this step? This is the number gradient descent actually optimises; more negative because $\beta$ amplifies the penalty.

Verify: units — reconstruction and KL are both in nats, so the objective is in nats: $-76$ nats. The bit figure $17.31$ satisfies $17.31/1.4427=12$ nats. $✓$

---

Example 8 — Cell H: the exam twist (recover the true log-likelihood)

Forecast: the parent's gap identity is $\log p_{\theta }(x)=\mathcal{L}+\text{gap}$. Guess the log-likelihood before computing.

1. Apply the gap identity. $\log p_{\theta }(x)=\mathcal{L}+D_{KL}(q\Vert p_{\theta }(z|x))=-95.0+4.0=-91.0$. Why this step? This is the exact decomposition from the parent — the gap is always a KL, hence $\ge 0$, hence ELBO $\le \log p_{\theta }(x)$.
2. Reconciliation check for part (b). New state: $\mathcal{L}=-92.0$, gap $=2.0$, so $\log p_{\theta }(x)=-92.0+2.0=-90.0$. Why this step? True log-likelihood rose from $-91.0$ to $-90.0$, and the approximation gap shrank from $4.0$ to $2.0$.
3. Answer. Yes — both $\theta$ (generative quality, higher $\log p_{\theta }(x)$) and $\varphi$ (tighter bound, smaller gap) improved. This is the dual role Expectation Maximization and VAE training play simultaneously.

Verify: consistency — in both states $\mathcal{L}=\log p_{\theta }(x)-\text{gap}$: $-91.0-4.0=-95.0✓$ and $-90.0-2.0=-92.0✓$. The gap is non-negative in both, so no impossibility. $✓$

---