Exercises — Variational Autoencoders (VAE) theory

• $(A)$ is the reconstruction term. It asks: after encoding $\mathbf{x}$ into a $\mathbf{z}$ and decoding it back, how likely is the original $\mathbf{x}$? Both the encoder $q_{\varphi }$ (it draws the $\mathbf{z}$) and the decoder $p_{\theta }$ (it scores $\mathbf{x}$) participate, but the score $\log p_{\theta }(\mathbf{x}|\mathbf{z})$ is owned by the decoder.
• $(B)$ is the regularization term (a Kullback-Leibler Divergence). It measures how far the encoder's output distribution is from the prior $p(\mathbf{z})=\mathcal{N}(0,\mathbf{I})$. It is owned by the encoder $q_{\varphi }$.

We subtract $(B)$ because we want to maximize $\mathcal{L}$: high reconstruction, low divergence from the prior.

The integral itself (the marginal likelihood, also called the evidence) is intractable. To evaluate it exactly you would sum the decoder's output over every latent code $\mathbf{z}$ in a high-dimensional space. There is no closed form, and grid-summing an $d$-dimensional space costs exponentially in $d$. This is precisely why we introduce the encoder $q_{\varphi }$ and the Evidence Lower Bound (ELBO) — we replace an impossible integral with a bound we can optimize by sampling.

(ii) is the Reparameterization Trick. In (i) the random draw sits inside a node that depends on $\varphi$, so there is no differentiable path from the loss back to ${\mathbf{\mu }}_{\varphi },{\mathbf{\sigma }}_{\varphi }$. In (ii) the only randomness is $\mathbf{ϵ}$, which does not depend on $\varphi$; ${\mathbf{\mu }}_{\varphi }$ and ${\mathbf{\sigma }}_{\varphi }$ enter through plain differentiable arithmetic (add, multiply), so backprop works.

Plug in ${\mu }^2=1$, ${\sigma }^2=4$, $\log {\sigma }^2=\log 4$: $D_{KL}=\frac{1}{2}\left(1+4-1-\log 4\right)=\frac{1}{2}\left(4-\log 4\right)=2-\frac{1}{2}\log 4.$ Numerically $\log 4=1.38629$, so $D_{KL}=2-0.69315=\boxed{1.30685}$. Look at figure s01: the wide, off-centre orange curve $\mathcal{N}(1,4)$ versus the teal standard prior — the KL penalty is the "cost" of that mismatch.

For a diagonal Gaussian the dimensions are independent, so the KL is the sum of per-dimension KLs: $D_{KL}={\sum }_{j=1}^3\frac{1}{2}\left({\mu }_j^2+{\sigma }_j^2-1-\log {\sigma }_j^2\right).$

• $j=1$: $\frac{1}{2}(0+1-1-\log 1)=\frac{1}{2}(0)=0$ (this dim already matches the prior).
• $j=2$: $\frac{1}{2}(0.25+1-1-\log 1)=\frac{1}{2}(0.25)=0.125$.
• $j=3$: ${\sigma }_3^2=0.25$, $\log 0.25=-1.38629$; $\frac{1}{2}(1+0.25-1+1.38629)=\frac{1}{2}(1.63629)=0.818146$. Total $=0+0.125+0.818146=\boxed{0.943146}$.

$z=\mu +\sigma ϵ=2+0.5\cdot (-1.2)=2-0.6=\boxed{1.4}$. The gradient $\partial z/\partial \mu =1$ and $\partial z/\partial \sigma =ϵ=-1.2$ — both finite, both flowing to $\varphi$. That is the whole point of the Reparameterization Trick.

$\frac{1}{2}({\mu }^2+{\sigma }^2-1-\log {\sigma }^2)=0$ has its minimum-and-zero at $\mu =0,{\sigma }^2=1$ (check: $0+1-1-\log 1=0$). So KL is zero exactly when the encoder outputs the prior itself, ignoring $\mathbf{x}$. This is the failure mode called Posterior Collapse: if a dimension always returns $\mathcal{N}(0,1)$ regardless of the input, it carries no information about $\mathbf{x}$. The decoder learns to ignore that latent. A zero KL term is therefore suspicious — it may mean the model found the lazy solution of matching the prior and reconstructing from nothing.

The ELBO is $\mathcal{L}=(\text{recon})-D_{KL}$. Hold the reconstruction expectation fixed. Since $D_{KL}\ge 0$ and enters with a minus, any increase $\Delta D_{KL}>0$ produces $\Delta \mathcal{L}=-\Delta D_{KL}<0$. So the KL acts as a pull toward the prior: the optimizer will never inflate KL for free. This is the "regularization vs reconstruction" balance from the parent note made precise — the two terms compete, and maximizing $\mathcal{L}$ trades them off.

The gap is $D_{KL}(q_{\varphi }\Vert p_{\theta }(\mathbf{z}|\mathbf{x}))$, which is $\ge 0$ and equals $0$ iff $q_{\varphi }(\mathbf{z}|\mathbf{x})=p_{\theta }(\mathbf{z}|\mathbf{x})$ almost everywhere. So the bound is tight exactly when the encoder recovers the true posterior. In practice a diagonal-Gaussian $q_{\varphi }$ usually cannot match a complex true posterior, so a residual gap remains — motivating richer families like Normalizing Flows. See figure s02: the log-evidence bar splits into the ELBO plus the posterior-approximation gap.

Minimizing $=-\mathcal{L}$. The reconstruction $\log p_{\theta }(\mathbf{x}|\mathbf{z})$ for independent Bernoulli pixels is the negative binary cross-entropy: $\log p_{\theta }(\mathbf{x}|\mathbf{z})={\sum }_k\left[x_k\log {\hat{x}}_k+(1-x_k)\log (1-{\hat{x}}_k)\right].$ The KL has the closed form from L2. With one reparameterized sample $\mathbf{z}=\mathbf{\mu }+\mathbf{\sigma }\odot \mathbf{ϵ}$: $\text{Loss}=-{\sum }_k\left[x_k\log {\hat{x}}_k+(1-x_k)\log (1-{\hat{x}}_k)\right]+\frac{1}{2}{\sum }_{j=1}^d\left({\mu }_j^2+{\sigma }_j^2-1-\log {\sigma }_j^2\right).$ What each piece is doing: first sum = "make the reconstruction match"; second sum = "keep the codes near the prior". We minimize their sum.

Reconstruction (negative BCE): $-[1\cdot \log 0.8+(1-1)\log (0.2)+0\cdot \log 0.3+(1-0)\log 0.7]=-[\log 0.8+\log 0.7].$ $\log 0.8=-0.223144$, $\log 0.7=-0.356675$; sum $=-0.579819$; negated $=0.579819$. KL: both ${\sigma }_j=1\Rightarrow \log {\sigma }_j^2=0$: $\frac{1}{2}[(0.25+1-1-0)+(0.25+1-1-0)]=\frac{1}{2}(0.25+0.25)=\mathrm{0.25.}$ Total loss $=0.579819+0.25=\boxed{0.829819}$.

Recon $=0.579819$ unchanged; scaled KL $=4\times 0.25=1.0$. Total $=0.579819+1.0=\boxed{1.579819}$. A larger $\beta$ pushes the codes harder toward the prior, trading reconstruction sharpness for more Disentangled Representations — each latent axis is pressured to independently match $\mathcal{N}(0,1)$.

Diagnosis: Posterior Collapse. The KL term dominated early, so the encoder took the cheapest route — output $\mathcal{N}(0,1)$ (KL $\to 0$, from Ex 3.1) — and the decoder learned to reconstruct from an uninformative code, leaving reconstruction stuck. Fixes:

1. KL annealing — start $\beta$ near $0$ and ramp it up, so the model first learns to reconstruct before regularizing.
2. Free bits / KL floor — allow each latent dim a small "free" nats budget of KL that isn't penalized, so useful dimensions aren't crushed to the prior. A stronger/autoregressive decoder makes this worse (it can reconstruct without $\mathbf{z}$), so weakening the decoder is a third option.

• (a) $\mathbf{z}=\mathbf{\mu }+\mathbf{\sigma }\odot \mathbf{ϵ}\to \mathbf{\mu }$: the sample becomes deterministic — a plain autoencoder.
• (b) KL per dim $=\frac{1}{2}({\mu }^2+{\sigma }^2-1-\log {\sigma }^2)$. As $\sigma \to 0$, $-\log {\sigma }^2\to +\infty$, so KL $\to +\infty$. The regularizer explodes, so the optimizer is strongly repelled from $\sigma =0$.
• (c) With near-zero variance the latent space is a set of isolated points with gaps between them (the very problem VAEs exist to prevent). Sampling a fresh $\mathbf{z}\sim \mathcal{N}(0,\mathbf{I})$ lands in an untrained gap → garbage generations. The infinite KL penalty is precisely nature's guardrail against this collapse to a deterministic autoencoder.

The VAE's whole objective is a tractable lower bound on log-likelihood (Bayesian Inference view): $\log p_{\theta }(\mathbf{x})\ge \mathcal{L}$. That bound requires $q_{\varphi }$ to be a genuine distribution so the expectation ${\mathbb{E}}_{q_{\varphi }}[\cdot ]$ and the KL $D_{KL}(q_{\varphi }\Vert p)$ are defined and the reparameterized gradient is unbiased. A single deterministic point has no density, so no ELBO, no likelihood interpretation, and no smoothing of the latent space. GANs abandon likelihood entirely for an adversarial game — a different trade-off (sharper samples, no explicit density, harder training). The VAE deliberately keeps the density so it can evaluate and bound $\log p_{\theta }(\mathbf{x})$.

One-word for the network approximating the posterior ::: encoder Sign in front of the KL term inside the ELBO ::: minus KL of $\mathcal{N}(1,4)$ from $\mathcal{N}(0,1)$ ::: 1.30685 Reparameterized $z$ with $\mu =2,\sigma =0.5,ϵ=-1.2$ ::: 1.4 Failure mode when KL collapses to zero ::: posterior collapse What happens to KL as $\sigma \to 0$ ::: it diverges to infinity