ELBO objective and KL term: Question bank — AI-ML deep dive

This page hunts the misconceptions around the ELBO objective and KL term. Every reveal below is a trap that a quick reader falls into. Try to answer before peeking — if your reasoning matches the "why", you actually understand the bound.

Before we start, here are the symbols we will lean on. Everything is defined here so this page stands on its own — no need to jump to the parent first.

Definition Cast of characters

$p_{θ} (x)$ — the evidence: the probability the generative model assigns to a real data point $x$ . We want it high but cannot compute it.
$q_{ϕ} (z ∣ x)$ — the encoder / approximate posterior: our controllable guess for "which latent codes $z$ likely produced this $x$ ". In VAEs it is a Gaussian $N (μ, diag (σ^{2}))$ whose mean $μ$ and standard deviations $σ$ are output by a neural network.
$p_{θ} (x ∣ z)$ — the decoder / likelihood: given a latent code $z$ , how probable the data point $x$ is. This is the network that turns codes back into data.
$p (z)$ — the prior over latent codes: what codes we believe in before seeing any data. In VAEs it is fixed to $N (0, I)$ (zero mean, unit variance in every dimension).
$p_{θ} (z ∣ x)$ — the true posterior: the exact (but intractable) answer to "which codes produced this $x$ ".

The picture below shows this split as heights on a bar — reconstruction pushes the bound up, KL pulls it down, and the two together can never rise above the true ceiling $lo g p_{θ} (x)$ .

The one equation that resolves half the traps: $lo g p_{θ} (x) = L (θ, ϕ; x) + D_{K L} (q_{ϕ} (z ∣ x) ∥ p_{θ} (z ∣ x))$ Read it as: true log-likelihood = the bound + the gap, and the gap is a KL divergence, so it can never be negative.

Intuition Where that master equation comes from — and

why the add-and-subtract trick works Start from Bayes' rule $p_{θ} (z ∣ x) = \frac{p _{θ} ( x ∣ z ) p ( z )}{p _{θ} ( x )}$ and take logs: $lo g p_{θ} (x) = lo g p_{θ} (x ∣ z) + lo g p (z) - lo g p_{θ} (z ∣ x)$ . This holds for every $z$ , so take the expectation over $z \sim q_{ϕ} (z ∣ x)$ (the left side has no $z$ , so it is unchanged).

Now the goal: we want to see two named KL divergences appear. Every KL divergence has the shape $E_{q} [lo g q - lo g (something)]$ — it always needs a $lo g q_{ϕ}$ inside. But our current expression has no $lo g q_{ϕ}$ term at all. So we manufacture one for free by adding $E_{q_{ϕ}} [lo g q_{ϕ} (z ∣ x)]$ and subtracting the same quantity (net change: zero, so the equation still holds).

Why that term specifically? Because it lets us pair up the logs into exactly the two KL shapes we are hunting for:

$(lo g p (z) - lo g q_{ϕ} (z ∣ x))$ averaged under $q_{ϕ}$ is $- D_{K L} (q_{ϕ} ∥ p)$ — pairs the prior with the encoder,
$(lo g q_{ϕ} (z ∣ x) - lo g p_{θ} (z ∣ x))$ averaged under $q_{ϕ}$ is $+ D_{K L} (q_{ϕ} ∥ p_{θ} (z ∣ x))$ — pairs the encoder with the true posterior.

The first pairing plus the leftover reconstruction $E_{q_{ϕ}} [lo g p_{θ} (x ∣ z)]$ is the ELBO; the second pairing is the gap. The trick works precisely because a $lo g q_{ϕ}$ is the missing ingredient both KL divergences need. The figure below traces this regrouping.

Finally, a question that trips up nearly everyone: why do we always write the KL as $q ∥ p$ (encoder first) and never $p ∥ q$ ? The next figure makes the difference geometric.

True or false — justify

The ELBO is always less than or equal to the true log-likelihood.

True. The gap term

D_{K L} (q_{ϕ} ∥ p_{θ} (z ∣ x))

is a KL divergence, which is

\geq 0

, so

L = lo g p_{θ} (x) - gap \leq lo g p_{θ} (x)

for every

ϕ

Maximizing the ELBO can never increase the true log-likelihood

lo g p_{θ} (x)

False. The ELBO depends on both

θ

and

ϕ

; changing

θ

moves the whole generative model, so climbing

L

genuinely raises

lo g p_{θ} (x)

— it is not just closing a fixed gap.

If the KL regularization term is zero, the encoder has failed.

False in general — it depends on which KL. If

D_{K L} (q_{ϕ} (z ∣ x) ∥ p (z)) = 0

the encoder outputs the prior regardless of

x

, which is posterior collapse and is a failure. But if the gap KL

D_{K L} (q_{ϕ} ∥ p_{θ} (z ∣ x)) = 0

, the encoder is perfect and the ELBO is tight — a triumph.

A larger ELBO always means better sample quality from the generator.

False. ELBO bundles reconstruction and KL; a model can score high ELBO while placing probability mass poorly for sampling, and Monte-Carlo ELBO estimates are also noisy — see Posterior Collapse for a case where ELBO looks fine but samples are useless.

KL divergence is a distance, so

D_{K L} (q ∥ p) = D_{K L} (p ∥ q)

False. KL is not symmetric and violates the triangle inequality, so it is not a metric. Swapping the arguments generally changes the value; VAEs deliberately use the reverse form

D_{K L} (q ∥ p)

The reconstruction term

E_{q_{ϕ}} [lo g p_{θ} (x ∣ z)]

can be positive.

True for continuous $x$ . A Gaussian density can exceed 1, so its log can be positive; only probabilities (discrete case) are capped at 1 and give non-positive logs.

Jensen's inequality is what guarantees the ELBO is a lower bound rather than upper.

True. Because

lo g

is concave,

lo g E [f] \geq E [lo g f]

, so pulling the log inside the expectation can only lose value — the result sits below

lo g p_{θ} (x)

q_{ϕ} (z ∣ x)

equals the true posterior

p_{θ} (z ∣ x)

, the ELBO equals the log-likelihood exactly.

True. Then the gap KL is zero, so

lo g p_{θ} (x) = L

— the bound is tight and Monte-Carlo estimation of

lo g p_{θ} (x)

becomes exact in expectation.

Spot the error

lo g p_{θ} (x) = lo g \int p_{θ} (x ∣ z) p (z) d z

is intractable, so we just replace the integral with

p_{θ} (x ∣ z_{0})

for one sampled

z_{0}

Error: one sample of

p_{θ} (x ∣ z_{0})

is not an unbiased estimate of the integral over the prior — most prior samples make

x

nearly impossible, so this estimate is astronomically high-variance. The ELBO fixes this by sampling from

q_{ϕ}

, which concentrates on the codes that actually explain

x

"Because

D_{K L} \geq 0

, dropping the KL term from the ELBO gives a tighter lower bound."

Error: removing the KL term does not tighten anything — it changes the objective entirely into just reconstruction, which is not a lower bound on

lo g p_{θ} (x)

and encourages posterior collapse in reverse (each

x

mapped to an arbitrary

z

"We use

D_{K L} (q_{ϕ} ∥ p_{θ} (z ∣ x))

inside the training loss and minimize it directly."

Error: we cannot compute

p_{θ} (z ∣ x)

(that was the whole problem), so we never touch the gap KL directly. The training loss uses

D_{K L} (q_{ϕ} (z ∣ x) ∥ p (z))

against the prior, and minimizing the gap happens implicitly by maximizing

L

"Jensen gives

f (E [X]) \leq E [f (X)]

, so

lo g E [f] \leq E [lo g f]

Error: the inequality direction is stated for convex

f

. Since

lo g

is concave, the inequality flips:

lo g E [f] \geq E [lo g f]

. Getting the direction wrong would make the "bound" an upper bound and break the whole derivation.

D_{K L} (q ∥ p) = \frac{1}{2} \sum_{j} (μ_{j}^{2} + σ_{j}^{2} - lo g σ_{j}^{2} - 1)

can be negative if the

σ_{j}

are tiny."

Error: the formula is provably

\geq 0

. As

σ_{j} \to 0

- lo g σ_{j}^{2} \to + \infty

dominates and drives KL up, not negative; the

- 1

is just the constant that makes KL

= 0

exactly at

μ = 0, σ = 1

"The expectation

E_{q_{ϕ}} [lo g p_{θ} (x ∣ z)]

is taken over the prior

p (z)

Error: it is taken over the encoder

q_{ϕ} (z ∣ x)

, not the prior. This is the point of variational inference — we sample the codes that

q_{ϕ}

says are relevant to this specific

x

, which is why the estimate is low-variance.

Why questions

Why do we pick the reverse KL

D_{K L} (q ∥ p)

instead of forward

D_{K L} (p ∥ q)

Because the expectation is over

q

, which is exactly the distribution we can sample from and control; forward KL would need samples from the intractable true posterior. This choice makes

q

zero-forcing — it avoids putting mass where

p

has none, at the cost of possibly ignoring some modes.

Why is the term called the Evidence Lower Bound and not just "the bound"?

In Bayesian language

p_{θ} (x)

is the evidence for the model, and

L

is a guaranteed lower bound on its logarithm — the name literally spells out "lower bound on the (log) evidence".

Why does the KL term prevent latent codes from drifting arbitrarily far apart?

It penalizes any encoder distribution that strays from the prior

p (z) = N (0, I)

, pulling every

q_{ϕ} (z ∣ x)

toward a common, overlapping region. This keeps the latent space continuous so that decoding a fresh

z \sim p (z)

produces sensible outputs — the sampling guarantee.

Why can we not just maximize

lo g p_{θ} (x)

directly if we have the decoder network?

Because

lo g p_{θ} (x) = lo g \int p_{θ} (x ∣ z) p (z) d z

integrates the decoder over all latent codes, an integral with no closed form and exponential cost in the latent dimension. The ELBO sidesteps this by replacing the integral with a tractable expectation under

q_{ϕ}

Why does the closed-form Gaussian KL make VAE training so much faster?

The KL term needs no Monte-Carlo sampling — it is an exact algebraic function of the encoder's

μ

and

σ

, so only the reconstruction term is estimated by sampling, cutting both cost and variance.

Why does raising a "

β

" weight on the KL term (as in

β

-VAE) risk posterior collapse?

Over-weighting KL makes the cheapest way to lower the loss be

q_{ϕ} (z ∣ x) \to p (z)

for all

x

, ignoring

x

entirely; the encoder stops carrying information — the collapse discussed in Posterior Collapse.

Edge cases

What is the ELBO if the encoder is deterministic, i.e.

σ_{j} \to 0

for all

j

The KL term

- lo g σ_{j}^{2} \to + \infty

blows up to

+ \infty

, so KL

\to + \infty

and the ELBO

\to - \infty

— the objective explicitly forbids a zero-variance encoder, which is why VAEs keep stochastic latents.

What happens to reconstruction and KL when

q_{ϕ} (z ∣ x) = p (z)

exactly for every

x

= 0

(perfect regularization) but the code carries no information about

x

, so reconstruction collapses to the best a blind decoder can do — this is textbook posterior collapse: great KL, useless model.

Is the ELBO still a valid lower bound if

q_{ϕ}

is a terrible approximation to the posterior?

Yes.

L \leq lo g p_{θ} (x)

holds for any valid distribution

q_{ϕ}

; a poor

q

just makes the gap KL large and the bound loose, but never invalid.

What does the ELBO gap look like at the very start of training with a randomly initialized encoder?

The gap

D_{K L} (q_{ϕ} ∥ p_{θ} (z ∣ x))

is typically huge, so

L

sits far below

lo g p_{θ} (x)

— training then simultaneously raises the ceiling (

θ

) and shrinks the gap (

ϕ

If a data point

x

is genuinely impossible under the model (

p_{θ} (x) \to 0

), what happens to

lo g p_{θ} (x)

and the ELBO?

lo g p_{θ} (x) \to - \infty

, and since

L \leq lo g p_{θ} (x)

the ELBO is dragged down with it — the bound faithfully reports "this point is unexplainable".

Can the gap KL and the regularization KL ever both be zero at once?

Only in the degenerate case where the true posterior equals the prior for that

x

(i.e.

z

tells you nothing about

x

). Then

q_{ϕ} = p = p_{θ} (z ∣ x)

, giving a tight bound but a latent code that ignores the data.

Does tightening the bound with more samples (importance weighting) change what "ELBO" means?

Yes — Importance Weighted Autoencoders use a multi-sample bound that is tighter (closer to

lo g p_{θ} (x)

) than the single-sample ELBO, though it is no longer decomposable into a clean reconstruction-minus-KL form.

Recall Self-check

Why is the gap between ELBO and log-likelihood never negative? ::: It equals $D_{K L} (q_{ϕ} (z ∣ x) ∥ p_{θ} (z ∣ x))$ , and every KL divergence is $\geq 0$ . Which distribution's samples power the reconstruction expectation? ::: The encoder $q_{ϕ} (z ∣ x)$ , not the prior $p (z)$ .