ELBO objective and KL term: Foundations — AI-ML deep dive

Before you can read a single line of the ELBO derivation, you need the alphabet it is written in. This page builds every symbol from nothing. Read it top to bottom: each idea is a brick for the next.

0. The cast of characters (what all the letters mean)

The parent note throws a lot of letters at you — $x$ , $z$ , $p$ , $q$ , $θ$ , $ϕ$ , $E$ , $D_{K L}$ . None of them are scary once you meet them one at a time.

Look at the figure. The hidden cause $z$ sits on the left. An arrow labelled "generate" pushes it to the visible data $x$ on the right. This one arrow is the whole idea of a generative model: nature (or our model) picks a hidden $z$ , then produces the $x$ we observe. Our job is to run that arrow backwards — see $x$ , guess $z$ .

1. Probability $p (\cdot)$ — the "how likely" machine

Why the topic needs it: everything in generative modelling is a question of "how likely." We measure success by how much probability the model assigns to real data.

Three flavours of $p$ appear, and mixing them up is the #1 source of confusion:

Notation	Read it as	Picture
$p (z)$	"how likely is this hidden cause, before seeing anything"	the prior — a fixed bell curve
$p (x ∥ z)$	"if the hidden cause is $z$ , how likely is this data"	the decoder — the generate arrow
$p (z ∥ x)$	"given I saw this data, how likely was each hidden cause"	the posterior — the reverse arrow

The little bar $∣$ means "given" — everything after the bar is assumed known.

2. The integral $\int \dots d z$ — "add up over all possibilities"

In the figure, the shaded area under the curve is the integral. Each thin vertical strip is one possible value of $z$ ; its height is how much that value contributes; adding every strip gives the whole area.

Why the topic needs it: to find how likely the data $x$ is overall, we must consider every hidden cause $z$ that could have produced it and add up their contributions: $p (x) = \int p (x ∣ z) p (z) d z$

Read this in words: "the total likelihood of $x$ = for every hidden cause $z$ , (how likely that cause is) $\times$ (how likely it makes $x$ ), summed over all $z$ ."

3. The logarithm $lo g$ — turning products into sums

Why the topic needs it: probabilities of many things multiply together and become vanishingly tiny (like $1 0^{- 500}$ ). Taking $lo g$ turns that fragile product into a stable sum, and because $lo g$ is increasing, maximizing $lo g p (x)$ is the same as maximizing $p (x)$ . So we always work with $lo g p (x)$ , never raw $p (x)$ .

Also, $lo g$ has a shape that matters enormously later:

The $lo g$ curve bulges upward (it is concave — it holds water). Notice in the figure: the straight chord between two points on the curve always sits below the curve. That single geometric fact is what makes Jensen's inequality (Step 3 of the derivation) point the right way. Hold this picture.

4. Expectation $E$ — the average you'd expect

Why the topic needs it: we can't do the full integral, but we can draw a handful of samples and average. Expectation is the bridge between the impossible integral and a doable computation (this is Monte Carlo — estimating an average by sampling).

5. The approximate posterior $q_{ϕ} (z ∣ x)$ — our stand-in for the impossible reverse arrow

The true reverse arrow $p (z ∣ x)$ needs the villain integral. So we build a replacement we can compute, and call it $q$ .

The letters $θ$ (theta) and $ϕ$ (phi) are just names for two sets of adjustable numbers:

$θ$ = knobs of the decoder ( $z \to x$ , the generate arrow).
$ϕ$ = knobs of the encoder ( $x \to z$ , the guess arrow).

6. KL divergence $D_{K L} (q ∥ p)$ — the distance between two clouds of belief

Read the figure: when the two bells sit on top of each other (left), $q / p \approx 1$ , its $lo g$ is $0$ , and KL $= 0$ . When they drift apart (right), the ratio $q / p$ swings away from 1 wherever $q$ has mass, the $lo g$ grows, and KL climbs. KL is small when the clouds overlap, large when they separate.

Two facts you must trust (proven on KL Divergence):

$D_{K L} \geq 0$ always — you can never be "negatively different."
$D_{K L} (q ∥ p) \neq = D_{K L} (p ∥ q)$ in general — the order matters. It is not an ordinary symmetric distance.

Why the topic needs it: the ELBO has a KL term that glues the encoder's guesses to the simple prior, and the gap between ELBO and the true log-likelihood is itself a KL. KL is the mortar of the whole building.

How it all feeds the topic

Every arrow entering ELBO is a symbol you now own. The impossible evidence at the top is exactly what the ELBO climbs toward from below.

Sanity check with one tiny number

Recall

Cloze yourself before moving on:

The letter that means hidden, unobserved cause

z

(latent variable)

The letter that means the data you can see

x

(observed)

p (z)

is called the

prior — belief about hidden causes before seeing data

p (x ∣ z)

is called the

decoder / likelihood — how a cause generates data

p (z ∣ x)

is called the

(true) posterior — the reverse arrow, usually intractable

Why is

p (x) = \int p (x ∣ z) p (z) d z

impossible in practice

the integral runs over a huge-dimensional

z

-space, more points than we can ever sum

Why do we take

lo g

of probabilities

it turns tiny products into stable sums and is increasing, so maximizing

lo g p

maximizes

p

The shape of

lo g

that makes Jensen work

concave (bulges up); chords lie below the curve

E_{z \sim q} [f (z)]

means

the average of

f (z)

when

z

is drawn from

q

The subscript

ϕ

q_{ϕ}

denotes

the encoder's tunable neural-network weights

D_{K L} (q ∥ p) = 0

exactly when

q

and

p

are the same distribution

D_{K L}

symmetric

no,

D_{K L} (q ∥ p) \neq = D_{K L} (p ∥ q)

in general

Equipment checklist

Say each answer out loud; if any stalls, re-read that section before the derivation page.

I can explain the difference between

x

and

z

in one sentence

x

is observed data,

z

is the hidden cause behind it

I can name all three:

p (z)

p (x ∣ z)

p (z ∣ x)

prior, decoder, posterior

I can say why

p (x)

is intractable

it needs an integral over all hidden causes in high dimensions

I know the two properties of

lo g

we use

lo g (ab) = lo g a + lo g b

, and

lo g

is increasing/concave

I can read

E_{z \sim q} [f (z)]

aloud and say what the subscript does

"average of

f

over

z

drawn from

q

"; the subscript fixes the sampling distribution

I know what

θ

and

ϕ

are

decoder weights and encoder weights

I can state what

D_{K L}

measures and its two key facts

mismatch between two distributions; always

\geq 0

, and not symmetric

I can compute a 1-D Gaussian KL from the closed form

\frac{1}{2} (μ^{2} + σ^{2} - lo g σ^{2} - 1)

Prerequisite deep-dives if any box failed: KL Divergence, Variational Inference, VAE Architecture. Related once you're solid: Expectation Maximization, Information Bottleneck, Posterior Collapse, Normalizing Flows, Importance Weighted Autoencoders. Then return to the parent: ELBO objective and KL term.