Latent diffusion (Stable Diffusion): Question bank

This page is a stress-test for your mental model of Latent diffusion (Stable Diffusion). Each item is a Question ::: Answer reveal. Read the question, commit to an answer out loud, then reveal. The answer side always gives a reason, never a bare yes/no — because the point is to repair the why, not just the what.

Before we start, three words we lean on constantly, in plain language:

True or false — justify

State true/false, then give the reason.

The VAE and the diffusion U-Net are trained together, end-to-end, in one loss.

False. The VAE is trained first and frozen; diffusion training then runs entirely on the frozen latents, so the U-Net never backprops into the encoder or decoder.

The 4 channels of a Stable Diffusion latent are red, green, blue, and alpha.

False. The 4 channels are learned abstract features, not colour channels — the VAE decides what each channel means during training. RGB only exists again after the decoder

D

Denoising happens in pixel space and the VAE is just a final polish step.

False. The whole reverse loop runs in latent space; the decoder

D

is called once at the very end to turn

z_{0}

into pixels. This is the entire point of "latent" diffusion.

A plain (non-variational) autoencoder would work just as well for the latent space.

False. Without the KL term the latent space develops "holes" that decode to garbage; a denoising trajectory passing through one breaks the image. The smoothness is a requirement, not a nicety.

The forward noising schedule

\overset{α}{ˉ}_{t} = \prod_{i = 1}^{t} (1 - β_{i})

is different from DDPM's.

False. The math of noising is identical to DDPM; the only change is that it acts on latents

z

instead of pixels

x

. Latent diffusion is DDPM in a compressed room.

Text conditioning enters through cross-attention where the image supplies

Q

and the text supplies

K, V

True. Each spatial patch of the latent forms a query

Q

and "looks up" which text tokens to attend to; the text embedding provides keys

K

and values

V

. That is how "sunglasses" reaches the right region.

Latent diffusion is inherently lower quality than pixel diffusion because it compresses.

False. The VAE is lossy but perceptually near-lossless — it throws away redundancy, not semantics. The saved compute buys more training, so latent diffusion matches or beats pixel diffusion in practice.

The classifier-free guidance scale changes what the VAE decoder does.

False. Classifier-Free Guidance reshapes the noise prediction inside the loop; the decoder is untouched and runs identically regardless of guidance strength.

Switching from DDPM sampling to DDIM Sampling requires retraining the U-Net.

False. Both samplers use the same trained

ϵ_{θ}

; DDIM just changes the update rule that turns a noise prediction into the next latent, so you can swap samplers at inference freely.

Spot the error

Each line contains a wrong statement. Reveal fixes it.

"The U-Net predicts the clean latent

z_{0}

directly."

It predicts the noise

ϵ

that was added, i.e.

ϵ_{θ} (z_{t}, t, c)

;

z_{0}

(or

z_{t - 1}

) is then computed from that prediction via the sampling formula.

"Because the latent is

64 \times 64

, the U-Net's self-attention cost drops by exactly

64 \times

Self-attention scales as

N^{2}

where

N = h \cdot w

, so cutting

N

64 \times

cuts attention cost by about

6 4^{2} \approx 4096 \times

; only the convolutional work drops linearly (

\sim 64 \times

"We compute

z_{T} = E (noise)

to start generation."

We sample

z_{T} \sim N (0, I)

directly; the encoder

E

is only used to make training latents from real images, never at generation start.

"The 512×512 → 64×64 downsampling uses factor

f = 4

The factor is

f = 8

(

512/8 = 64

);

f = 4

would give

128 \times 128

"CLIP's embedding

c

is a single 768-vector for the whole prompt."

It is a

77 \times 768

matrix — one 768-vector per token slot — which is precisely what lets different regions attend to different words.

"The KL term in the VAE makes the reconstruction pixel-perfect."

KL does the opposite of that — it regularizes the latent toward a smooth Gaussian-like shape (aiding diffusion); reconstruction fidelity comes from the L2 pixel loss, and it is deliberately not pixel-perfect.

"Attention cost scaling as

N^{2}

means doubling the image side doubles the attention cost."

Doubling the side quadruples

N

(area), and

N^{2}

then rises

16 \times

— the quadratic blow-up in area is exactly why pixel-space attention was the bottleneck.

"Since the VAE is frozen, its choice doesn't affect final image quality."

A frozen VAE still sets a hard ceiling: diffusion can never recover detail the decoder cannot render, so a weak VAE caps quality no matter how good the U-Net is.

Why questions

Answer the why, not just the fact.

Why use a variational autoencoder instead of a plain one for the latent space?

The KL term forces a continuous, smooth latent space where nearby latents decode to similar images; diffusion moves gradually through latent space and needs this continuity to avoid landing in decode-to-garbage holes.

Why denoise latents instead of pixels at all?

Natural images are hugely redundant (neighbouring pixels correlate); working in the

\sim 48 \times

-smaller latent removes that redundancy, so compute focuses on semantics instead of fighting dimensionality — dramatically cheaper attention.

Why does the U-Net predict noise

ϵ

rather than the clean signal?

Predicting

ϵ

gives a well-scaled, roughly unit-variance target at every timestep, which trains stably; the clean latent can then be algebraically recovered from

z_{t}

and

\overset{ϵ}{^}

when needed.

Why is cross-attention (not just concatenation) used to inject text?

Attention lets each spatial patch independently choose which text tokens matter to it, so "spacesuit" influences the body region and "corgi" the face — a fixed concatenation could not route information region-by-region.

Why can we freeze the VAE and still train diffusion successfully?

The VAE defines a fixed, stable target manifold of "valid latents"; diffusion only needs to learn to navigate that fixed space, so re-training the VAE mid-way would just move the goalposts and destabilize learning.

Why does the reverse step multiply the bracket by

\frac{1}{α _{t}}

After subtracting the predicted noise the remaining signal is shrunk (because forward noising scaled

z_{0}

down); the

\frac{1}{α _{t}}

factor rescales it back to the correct magnitude for the next step.

Why add fresh noise

σ_{t} z

during DDPM sampling if we're trying to remove noise?

A little re-injected randomness restores diversity and matches the true reverse distribution's variance; sampling with

σ_{t} = 0

(as DDIM Sampling can) is deterministic but yields less variety per seed.

Edge cases

Boundary and degenerate scenarios.

What is

\overset{α}{ˉ}_{t}

t = T

(the last forward step) and what does that mean for

z_{T}

\overset{α}{ˉ}_{T} \approx 0

, so

z_{T} \approx 1 - \overset{α}{ˉ}_{T} ϵ \approx ϵ

— the latent is essentially pure Gaussian noise, which is exactly why generation can start from random noise.

What happens at

t = 0

in the forward process?

\overset{α}{ˉ}_{0} \approx 1

, giving

z_{0} \approx z_{0}

with negligible noise — the clean encoded latent, the endpoint the reverse loop is trying to reach.

If you set the classifier-free guidance scale to

1

, what conditioning behaviour do you get?

Guidance scale

1

means no amplification — you get the plain conditional prediction, so prompts are followed weakly; higher scales exaggerate the difference between conditional and unconditional predictions.

What if the input image contains fine text or a barcode — does the VAE handle it?

Poorly; those are exactly the high-frequency, non-redundant details the compression struggles to preserve, which is why generated small text is often garbled — a genuine limitation of the latent bottleneck.

What if you feed the U-Net an empty prompt (embedding of ""), what does it produce?

The unconditional prediction — a plausible but unguided image; this same empty-prompt path is precisely what Classifier-Free Guidance uses as its "unconditional" branch.

If the noise schedule made

β_{i} = 0

for all

i

, what breaks?

Then

\overset{α}{ˉ}_{t} = 1

for every

t

, so no noise is ever added,

z_{T} = z_{0}

, and there is nothing to denoise — the model can learn no reverse process and generation is impossible.

At extremely high sampling steps (

T

very large), what is the trade-off?

Quality gains flatten while compute grows linearly with steps; beyond a point (often ~50 with DDIM) extra steps mostly waste time, which is why few-step samplers exist.

Can the same trained U-Net denoise a latent of a different spatial size (e.g.

96 \times 96

Partly — convolutions and attention are size-agnostic in principle, but the model was trained at one resolution, so off-size latents often show repetition or artifacts; it works but degrades outside the training regime.

Recall Quick self-test

Fresh-noise-during-denoising purpose ::: restores diversity / matches reverse-process variance; $σ_{t} = 0$ makes it deterministic. What the 4 latent channels are ::: learned abstract features, not RGB. Why VAE not plain AE ::: KL keeps latent smooth/continuous so diffusion never hits decode-to-garbage holes. Attention cost scaling in $N = h \cdot w$ ::: quadratic, $N^{2}$ — the reason pixel-space was the bottleneck.