Variational Autoencoders (VAE) theory: Question bank

This page is a set of conceptual traps: short questions built around the mistakes people actually make with VAEs. Each answer is real reasoning, not a bare yes/no. Cover the reveals one at a time and argue your case before you peek.

Reminder of the symbols (all defined in the parent note):

$x$ = a data point (e.g. an image), $z$ = a latent code (a short list of numbers).
$p (z) = N (0, I)$ = the prior, a standard bell curve centred at zero with spread one.
$q_{ϕ} (z ∣ x)$ = the encoder distribution, with mean $μ_{ϕ} (x)$ and variance $σ_{ϕ}^{2} (x)$ .
$L$ = the ELBO, the quantity we maximize.
$D_{K L}$ = KL divergence, a one-directional "distance" between distributions.

🇮🇳 Hinglish version of the parent: yahaan padho.

True or false — justify

TF1. "Maximizing the ELBO is exactly the same as maximizing

lo g p_{θ} (x)

False. They differ by

D_{K L} (q_{ϕ} ∥ p_{θ} (z ∣ x))

, which is

\geq 0

; the ELBO equals the log-likelihood only when the encoder matches the true posterior exactly, which almost never happens.

TF2. "A VAE with a perfect reconstruction loss of zero is the ideal VAE."

False. Zero reconstruction error usually means the encoder collapsed variance to near zero and behaves like a deterministic autoencoder; the latent space then has gaps and sampling from the prior gives garbage.

TF3. "The KL term

D_{K L} (q_{ϕ} (z ∣ x) ∥ p (z))

is symmetric, so we could write it the other way round without changing anything."

False. KL divergence is not symmetric;

D_{K L} (q ∥ p) \neq = D_{K L} (p ∥ q)

in general. VAEs specifically use the "reverse" form

q ∥ p

, which is mode-seeking and lets

q

ignore regions where

p

has mass.

TF4. "The reparameterization trick changes the distribution of

z

False. Writing

z = μ + σ ⊙ ϵ

with

ϵ \sim N (0, I)

produces the exact same Gaussian

N (μ, σ^{2})

; it only relocates the randomness so gradients can flow through

μ

and

σ

TF5. "Because

p (z) = N (0, I)

, every individual encoded point

q_{ϕ} (z ∣ x)

must also be centred at zero."

False. The aggregate of all encoded points is pushed toward

N (0, I)

, but each single

q_{ϕ} (z ∣ x)

has its own mean

μ_{ϕ} (x)

that can sit far from zero — that spread is how the model distinguishes data points.

TF6. "If the KL term becomes exactly zero for a data point, the model is doing great on that point."

False.

D_{K L} = 0

means

q_{ϕ} (z ∣ x) = p (z)

exactly — the encoder ignored

x

entirely. This is posterior collapse and the decoder gets no useful information.

TF7. "The ELBO is a lower bound, so making it larger can never hurt the true log-likelihood."

True in spirit, but subtle. Raising the ELBO raises the floor under

lo g p_{θ} (x)

, but the two can move apart if the posterior gap

D_{K L} (q_{ϕ} ∥ p_{θ} (z ∣ x))

grows; you push a bound up, you do not directly observe the true value.

TF8. "A GAN and a VAE optimize the same objective, just with different networks."

False. A VAE maximizes a likelihood lower bound (ELBO); a GAN plays a minimax game against a discriminator with no explicit likelihood. This is why VAE samples are often blurrier but its latent space is smoother and more probabilistic.

Spot the error

SE1. "We predict

σ^{2}

directly from the encoder because variance is what the formula needs."

Error: we predict

lo g σ^{2}

, not

σ^{2}

. A raw network output can be negative, but variance must be positive;

σ^{2} = exp (lo g σ^{2})

guarantees positivity and is numerically stable.

SE2. "The integral

p_{θ} (x) = \int p_{θ} (x ∣ z) p (z) d z

is intractable because the decoder is nonlinear, so we just approximate the decoder with a linear model."

Error: the intractability comes from integrating over a high-dimensional

z

, not from decoder nonlinearity. The fix is variational inference (introduce

q_{ϕ}

), not linearizing the decoder.

SE3. "In the ELBO derivation we applied Jensen's inequality as

lo g E [X] \leq E [lo g X]

Error: the inequality goes the other way for the concave

lo g

lo g E [X] \geq E [lo g X]

. That direction is exactly what makes

L

a lower bound.

SE4. "We can just sample

z \sim N (μ_{ϕ}, σ_{ϕ}^{2})

and backpropagate through the sampling node."

Error: a sampling operation is not differentiable with respect to

μ_{ϕ}, σ_{ϕ}

. You must apply the reparameterization trick to move randomness into an external

ϵ

SE5. "The reconstruction term

E_{q_{ϕ}} [lo g p_{θ} (x ∣ z)]

measures how close

z

is to the prior."

Error: that term measures how well the decoder rebuilds

x

from

z

. Closeness to the prior is measured by the KL term

D_{K L} (q_{ϕ} ∥ p)

SE6. "Since

D_{K L} \geq 0

always, we can drop it from the loss to speed up training."

Error: dropping the KL term removes the pressure that shapes the latent space into the prior. Without it you get an ordinary autoencoder with a disorganized latent space you cannot sample from.

SE7. "The closed-form Gaussian KL uses

E_{q} [z_{j}^{2}] = σ_{j}^{2}

Error: for

z_{j} \sim N (μ_{j}, σ_{j}^{2})

E_{q} [z_{j}^{2}] = μ_{j}^{2} + σ_{j}^{2}

(variance plus mean-squared). Forgetting the

μ_{j}^{2}

term breaks the derivation.

SE8. "Using more Monte Carlo samples per data point changes the objective we are optimizing."

Error: more samples reduce the variance of the gradient estimate but the expected objective is the same; VAEs commonly use just one sample per point.

Why questions

WHY1. Why do we introduce an encoder

q_{ϕ} (z ∣ x)

at all, instead of using the true posterior

p_{θ} (z ∣ x)

The true posterior requires

p_{θ} (x)

in its denominator (via Bayes' rule), and that marginal is the very intractable integral we cannot compute.

q_{ϕ}

is a cheap, learnable approximation that sidesteps it.

WHY2. Why choose a Gaussian for the prior rather than, say, a uniform distribution?

A Gaussian has infinite support (no hard edges to fall off), a simple closed-form KL against a Gaussian encoder, and is smooth so nearby codes decode to nearby data — properties that make sampling well-behaved.

WHY3. Why is the KL term called a "regularizer"?

It penalizes the encoder for placing codes far from the prior, pulling all encodings toward one shared shape. This prevents the latent space from fragmenting into isolated islands, just as an L2 penalty prevents weights from blowing up.

WHY4. Why does maximizing the ELBO indirectly shrink the posterior gap?

Because

lo g p_{θ} (x) = L + D_{K L} (q_{ϕ} ∥ p_{θ} (z ∣ x))

and the left side is fixed for fixed

θ

; raising

L

(by tuning

ϕ

) must lower the gap term, forcing

q_{ϕ}

closer to the true posterior.

WHY5. Why do VAE samples often look blurrier than GAN samples?

The reconstruction term is an average likelihood over the encoder distribution, and with a Gaussian decoder it rewards predicting the mean of plausible outputs, which averages out sharp high-frequency detail into smooth blur.

WHY6. Why does the

β

-VAE (weighting the KL term by

β > 1

) encourage disentangled representations?

A stronger KL pressure forces the encoder to use latent dimensions frugally and near-independently to stay close to the factorized prior, which tends to align individual latent axes with independent factors of variation.

WHY7. Why is the reparameterization trick specifically easy for Gaussians but harder for discrete latents?

A Gaussian is a smooth, invertible transform of

ϵ

(

z = μ + σ ⊙ ϵ

), so gradients pass cleanly. Discrete draws have no such differentiable transform, requiring workarounds like the Gumbel-softmax relaxation.

WHY8. Why can a very powerful, expressive decoder actually cause posterior collapse?

If the decoder can model

x

well on its own (e.g. an autoregressive decoder), the model can drive the KL term to zero and ignore

z

entirely, because it no longer needs the latent information to reconstruct.

Edge cases

EC1. What happens to a single encoder distribution

q_{ϕ} (z ∣ x)

as its predicted variance

σ_{ϕ}^{2} \to 0

It becomes a spike (a point mass) at

μ_{ϕ}

, so the VAE degenerates into a deterministic autoencoder for that point; the KL term also blows up because a near-delta is far from the spread-out prior.

EC2. What does the loss do when

μ_{ϕ} (x) = 0

and

σ_{ϕ}^{2} (x) = 1

for every

x

The KL term is exactly zero everywhere, but the encoder carries no information about

x

— the textbook posterior collapse failure where reconstruction must be handled by the decoder alone.

EC3. If you feed the trained decoder a latent point with a huge norm, say

∥ z ∥ = 20

, what should you expect?

Since the prior

N (0, I)

almost never produces such extreme codes, the decoder was rarely trained there and will likely output an unrealistic or degenerate sample — the latent space is only reliable near the origin.

EC4. What is the value of

D_{K L} (q ∥ p)

when

q = p

exactly?

Exactly zero, because

lo g (q / p) = lo g 1 = 0

everywhere. KL is zero iff the two distributions are identical, and is strictly positive otherwise.

EC5. In the limit where the decoder outputs a fixed image regardless of

z

, what happens to the reconstruction term?

It flatlines at the likelihood of that one fixed output; it cannot improve by using

z

, so the model has no incentive to keep

z

informative — another route into collapse.

EC6. What happens to the ELBO's two terms if the KL weight is set to

β = 0

The regularizer vanishes and only reconstruction remains, so you recover a plain autoencoder: great reconstructions but an unstructured latent space you cannot sample from meaningfully.

EC7. If the true posterior

p_{θ} (z ∣ x)

happens to be non-Gaussian (multi-modal), what limits a standard VAE encoder?

A Gaussian

q_{ϕ}

can only capture one mode with a single bump, so the posterior gap stays large; richer families like normalizing flows for

q_{ϕ}

are used to close it.

EC8. For a conditional VAE, what changes in the prior when we condition on a label

y

Everything becomes conditioned on

y

: the encoder is

q_{ϕ} (z ∣ x, y)

and the prior/decoder use

y

too, so sampling can be steered to produce data of a chosen class rather than a random one.

Recall Quick self-test

The three-way identity $lo g p_{θ} (x) = L + D_{K L} (q_{ϕ} ∥ p_{θ} (z ∣ x))$ — which term is fixed and which two trade off? ::: For fixed $θ$ , $lo g p_{θ} (x)$ is fixed; raising the ELBO $L$ must shrink the posterior gap. Which single failure mode do TF6, EC2 and WHY8 all describe? ::: Posterior collapse — the encoder stops using $x$ and the KL term hits zero.