Worked examples — Variational Autoencoders (VAE) theory

This page is a hands-on companion to the VAE theory note. We will compute the pieces that the parent note only wrote as symbols. Everything here reduces to two numbers you can check by hand: the reconstruction term and the KL term, whose sum (with a minus sign) is the ELBO we maximise.

Before any example, let me re-earn every symbol so a newcomer can read line one.

Why this exact formula and not a general integral? Because both blobs are Gaussians, the messy integral from the parent note collapses to elementary arithmetic on $\mu$ and $\sigma$. That is the reason VAEs use Gaussians — the regulariser becomes a formula you can differentiate and even evaluate by hand.

---

The scenario matrix

Every VAE computation you will meet lives in one of these cells. The examples below are labelled by cell so you can see we skipped nothing.

Cell	Scenario	What is stressed
C1	$\mu =0,\sigma =1$ (encoder already matches prior)	KL should be exactly $0$ (degenerate/ideal case)
C2	$\mu \ne 0,\sigma =1$ (shifted, right width)	only the ${\mu }^2$ force acts
C3	$\mu =0,\sigma \ne 1$ — both $\sigma >1$ and $\sigma <1$	the ${\sigma }^2-1-\log {\sigma }^2$ force, both signs
C4	Limiting case $\sigma \to 0$ (blob collapses to a point)	KL $\to \infty$; why a deterministic encoder is punished
C5	Multi-dimensional latent, mixed dimensions	KL is a sum; Posterior Collapse preview
C6	Reparameterization: turn $ϵ$ into a concrete $z$	separating randomness from parameters
C7	Full ELBO on a tiny Bernoulli pixel (word problem)	reconstruction + KL together, real units (nats)
C8	Exam twist: $\beta$-VAE / which term dominates	reasoning about the trade-off, no arithmetic trap

---

Worked examples

C1 — The perfect encoder (degenerate baseline)

Forecast: Guess before reading. The encoder's blob is the target. What should the "distance" be?

1. Write the per-dimension formula. With $d=1$: $D_{KL}=\frac{1}{2}\left({\mu }^2+{\sigma }^2-1-\log {\sigma }^2\right)$ Why this step? Both distributions are Gaussian, so we may use the closed form instead of integrating.
2. Substitute $\mu =0,{\sigma }^2=1$. $D_{KL}=\frac{1}{2}\left(0+1-1-\log 1\right)$ Why this step? $\log 1=0$, so every term vanishes.
3. Evaluate: $D_{KL}=\frac{1}{2}(0)=0$.

Verify: KL divergence between two identical distributions is always $0$, and $\ge 0$ otherwise. Our result sits exactly at the floor — consistent. Units: nats (natural-log information units).

---

C2 — Shifted mean, correct width

Forecast: Only the centre moved. Which of the three forces should switch on?

1. Formula, ${\sigma }^2=1$: $D_{KL}=\frac{1}{2}\left({\mu }^2+1-1-\log 1\right)=\frac{1}{2}{\mu }^2$. Why this step? With correct width the $\sigma$-forces cancel exactly, isolating the mean penalty.
2. Plug $\mu =2$: $D_{KL}=\frac{1}{2}(4)=2$ nats. Why this step? ${\mu }^2=4$; the penalty grows quadratically with drift — moving twice as far costs four times as much.

Verify: Sign check — KL is positive (the blob no longer matches the prior). Symmetry check: $\mu =-2$ would give the same $2$ nats, because the formula uses ${\mu }^2$; picture two blobs shifted equally left and right of centre — they are equally "wrong". ✓

---

C3 — Wrong width, both directions

Here we split into the two sub-cases the matrix demands: too wide ($\sigma >1$) and too narrow ($\sigma <1$). See the figure — it plots the width-penalty $f({\sigma }^2)=\frac{1}{2}({\sigma }^2-1-\log {\sigma }^2)$.

Look at the green curve: it bottoms out at exactly ${\sigma }^2=1$ (dashed white line) with value $0$, and rises on both sides. That is why any deviation of width, wider or narrower, costs KL.

Forecast: Which is punished harder — a blob twice as wide, or one half as wide?

1. Formula with $\mu =0$: $D_{KL}=\frac{1}{2}\left({\sigma }^2-1-\log {\sigma }^2\right)$. Why this step? Zero mean kills the ${\mu }^2$ term, leaving the pure width force.
2. C3a, ${\sigma }^2=4$: $\frac{1}{2}(4-1-\log 4)=\frac{1}{2}(3-1.3863)=\frac{1}{2}(1.6137)\approx 0.8069$ nats. Why this step? $\log 4\approx 1.3863$; the linear ${\sigma }^2$ growth dominates for wide blobs.
3. C3b, ${\sigma }^2=0.25$: $\frac{1}{2}(0.25-1-\log 0.25)=\frac{1}{2}(-0.75+1.3863)=\frac{1}{2}(0.6363)\approx 0.3181$ nats. Why this step? Now $\log {\sigma }^2$ is negative, so $-\log {\sigma }^2$ is a large positive penalty — this is the force that stops the blob shrinking to a point.

Verify: Both positive ✓. The too-wide blob ($0.807$) costs more than this particular too-narrow one ($0.318$), matching the curve's steeper right arm in the figure. As a limiting sanity check, C4 next shows the narrow arm eventually wins as $\sigma \to 0$. ✓

---

C4 — The collapse limit $\sigma \to 0$

Forecast: A standard autoencoder outputs a single point (zero width). Guess whether the VAE's KL rewards or punishes that.

1. Formula: $D_{KL}=\frac{1}{2}({\sigma }^2-1-\log {\sigma }^2)$.
2. Evaluate the three values:
   • ${\sigma }^2=0.1:\frac{1}{2}(0.1-1-\log 0.1)=\frac{1}{2}(-0.9+2.3026)\approx 0.7013$
   • ${\sigma }^2=0.01:\frac{1}{2}(0.01-1-\log 0.01)=\frac{1}{2}(-0.99+4.6052)\approx 1.8076$
   • ${\sigma }^2=0.001:\frac{1}{2}(0.001-1-\log 0.001)=\frac{1}{2}(-0.999+6.9078)\approx 2.9544$ Why this step? Each tenfold shrink adds about $\frac{1}{2}\log 10\approx 1.15$ nats — the penalty grows without bound.
3. Limit: as ${\sigma }^2\to 0$, the term $-\log {\sigma }^2\to +\infty$, so $D_{KL}\to +\infty$. Why this step? $\log$ of a number approaching zero dives to $-\infty$; negating it launches the penalty to $+\infty$.

Verify: The increments $0.701\to 1.808\to 2.954$ rise by $\approx 1.107,1.147$ — approaching the predicted $\frac{1}{2}\ln 10\approx 1.1513$. ✓

---

C5 — Multi-dimensional latent, mixed dimensions

Forecast: Which single dimension contributes nothing?

1. KL is a sum over independent dimensions: $D_{KL}={\sum }_{j=1}^3\frac{1}{2}({\mu }_j^2+{\sigma }_j^2-1-\log {\sigma }_j^2)$ Why this step? A diagonal Gaussian factorises, so information distances add — no cross terms.
2. Dimension 1 ($\mu =0,{\sigma }^2=1$): $\frac{1}{2}(0+1-1-0)=0$. (This is a "collapsed" dimension — a preview of Posterior Collapse.)
3. Dimension 2 ($\mu =1,{\sigma }^2=1$): $\frac{1}{2}(1+1-1-0)=0.5$.
4. Dimension 3 ($\mu =0,{\sigma }^2=0.25$): reuse C3b $=0.3181$.
5. Add: $0+0.5+0.3181=0.8181$ nats.

Verify: Total is the plain sum of three non-negative per-dim values ✓. Dimension 1 contributes exactly $0$ — an inactive latent the decoder ignores, precisely what posterior collapse looks like at the arithmetic level. ✓

---

C6 — Reparameterization: from $ϵ$ to a concrete $z$

Forecast: Which parts of the calculation depend on the trainable parameters, and which is "frozen noise"?

1. Apply the trick $z=\mu +\sigma ϵ$: $z=3+2\times 0.5=4.$ Why this step? This is the Reparameterization Trick: it moves the randomness out of the network into $ϵ$, so the path $\mu ,\sigma \to z$ is a plain differentiable formula.
2. Gradient check (by hand): $\frac{\partial z}{\partial \mu }=1,\frac{\partial z}{\partial \sigma }=ϵ=\mathrm{0.5.}$ Why this step? Both derivatives exist and are finite — that is the whole point. Sampling directly from $\mathcal{N}(3,4)$ would have no usable derivative w.r.t. $\mu ,\sigma$.

The figure shows it visually: fixed noise $ϵ$ on a standard bell curve, stretched by $\sigma$ and shifted by $\mu$ into the output blob.

Verify: Statistically, $z=\mu +\sigma ϵ$ with $ϵ\sim \mathcal{N}(0,1)$ has mean $\mu =3$ and variance ${\sigma }^2=4$ — exactly the distribution we wanted to sample from. Sanity: $\partial z/\partial \mu =1$ means a unit shift of the mean shifts the sample one-for-one. ✓

---

C7 — Full ELBO on a single Bernoulli pixel (word problem)

Forecast: The ELBO has two parts. Guess which one is a familiar cross-entropy.

1. Reconstruction term — log-likelihood of a Bernoulli pixel: $\log p_{\theta }(x\mid z)=x\log \hat{p}+(1-x)\log (1-\hat{p}).$ Why this step? A binary pixel is a coin flip; its log-probability is the Bernoulli log-likelihood (this is exactly negative binary cross-entropy).
2. Plug $x=1,\hat{p}=0.8$: $\log p=1\cdot \log 0.8+0=\log 0.8\approx -0.2231$ nats. Why this step? Correct-but-uncertain prediction gives a small negative log-likelihood; a perfect $\hat{p}=1$ would give $0$.
3. KL term ($\mu =0.5,{\sigma }^2=1$): $\frac{1}{2}(0.25+1-1-0)=0.125$ nats. (reuses the C2 pattern) Why this step? Regularises the encoder toward the prior; here only the small mean-drift acts.
4. ELBO $=\mathbb{E}[\log p_{\theta }(x\mid z)]-D_{KL}=-0.2231-0.125=-0.3481$ nats. Why this step? Definition of the ELBO from the parent note; we maximise it, i.e. minimise its negative.
5. Training loss $=-\text{ELBO}=0.3481$ nats.

Verify: Units all in nats ✓. Both pieces have the right sign: reconstruction log-likelihood $\le 0$ ✓, KL $\ge 0$ ✓. If the decoder were perfect ($\hat{p}=1$) and encoder ideal ($\mu =0,\sigma =1$), loss would drop to $0$ — the global optimum. ✓

---

C8 — Exam twist: which term wins in $\beta$-VAE?

Forecast: Big $\beta$ favours organisation over fidelity — guess which failure mode it invites.

1. Balance condition: set $\beta D_{KL}=|\log p|$, i.e. $\beta \times 0.125=0.2231$. Why this step? We want the point where the two competing forces are equal in size.
2. Solve: $\beta =0.2231/0.125\approx 1.7847$. Why this step? Above this, regularisation outweighs reconstruction.
3. Limit reasoning ($\beta \to \infty$): the loss is dominated by $\beta D_{KL}$, which is minimised at $D_{KL}=0$, i.e. $\mu =0,\sigma =1$ — the encoder ignores $x$ entirely. That is Posterior Collapse; it also links to why disentanglement is sought by moderate $\beta$, not extreme. Why this step? No arithmetic trap — the exam tests whether you can reason about the trade-off's endpoints.

Verify: At $\beta \approx 1.785$, $\beta D_{KL}=1.785\times 0.125\approx 0.2231=|\log p|$ ✓. As $\beta \to \infty$ the reconstruction term becomes negligible relative to the exploding KL weight, forcing $q$ onto the prior. ✓

---

Recall Quick self-test

What is $D_{KL}$ when $\mu =0,\sigma =1$? ::: Exactly $0$ — the encoder already matches the prior. Why does $\sigma \to 0$ blow up the KL? ::: The $-\log {\sigma }^2$ term goes to $+\infty$, forbidding a deterministic encoder. In the reparameterization $z=\mu +\sigma ϵ$, what is $\partial z/\partial \mu$? ::: $1$ — gradients flow straight through the mean. For a 3-D latent, how do the per-dimension KLs combine? ::: They add (the diagonal Gaussian factorises). As $\beta \to \infty$ in a $\beta$-VAE, what failure appears? ::: Posterior collapse ($\mu \to 0,\sigma \to 1$, latent ignored).

See also: Bayesian Inference for the posterior these examples approximate, and Generative Adversarial Networks (GANs) / Normalizing Flows / Conditional VAE for alternative generative objectives.