Foundations — Latent diffusion (Stable Diffusion)

Before you can read the parent note, you need to own every symbol it throws at you. Below, each symbol is introduced only after the one it depends on. Nothing is used before it is built.

1. Pixels, images, and the number $x$

Look at the figure: a 4×4 image is really $4\times 4\times 3=48$ numbers. A real 512×512 image is $512\times 512\times 3=786,432$ numbers.

• Plain words: $x$ is "the whole image, written as one long list of brightness numbers."
• Picture: a stack of three grids (Red grid, Green grid, Blue grid).
• Why the topic needs it: the entire goal is to produce an $x$ — a valid image — from noise.

Why this notation and not "a picture"? Because the network only ever sees numbers. Writing $x\in {\mathbb{R}}^{H\times W\times 3}$ tells us exactly how many numbers — and that count is the whole efficiency story of Latent diffusion (Stable Diffusion).

2. Dimension, and why "big" is expensive

• Picture: imagine each number as one lever on a giant control panel. Dimension = number of levers you must set.
• Why the topic needs it: the parent's headline "98% dimension reduction" is meaningless until you feel that dimension = money and time.

3. The latent $z$ — a compressed sketchpad

• Plain words: $z$ is the small drawing; $x$ is the big finished painting.
• Picture: a big 512×512 canvas shrinking down to a 64×64 pad of 4 abstract channels (not red/green/blue — learned features).
• Why the topic needs it: all the expensive denoising happens on $z$, not $x$.

The symbol $\ll$ means "much less than." We use it (not just $<$) to stress the gap is huge — that gap is where the speedup lives.

4. The autoencoder: $\mathcal{E}$ and $\mathcal{D}$

To move between the big painting $x$ and the small sketch $z$, we need two machines.

• Picture: a funnel narrowing (encode) then a funnel widening (decode), $x\to z\to \hat{x}$.
• Why we need both: we live in pixel-world (the prompt-giver wants a picture) but work in latent-world. $\mathcal{E}$ gets us in, $\mathcal{D}$ gets us out.

Why calligraphic letters $\mathcal{E},\mathcal{D}$? Just a convention that says "this is a whole neural network, not a single number." See Variational Autoencoders (VAE) for why the "V" (variational) version is required — it keeps $z$-space smooth with no holes.

5. Gaussian noise and $\mathcal{N}(0,I)$

• Picture: a bell curve, most darts landing near the center.
• Why the topic needs it: diffusion is entirely about noise — adding it (forward) and predicting it to remove it (reverse). If you don't know what $ϵ\sim \mathcal{N}(0,I)$ means, no formula in the parent parses.

6. The noisy latent $z_t$, timestep $t$, and the schedule ${\bar{\alpha }}_t$

The bar over $\alpha$ means "cumulative product from step 1 to $t$": ${\bar{\alpha }}_t={\prod }_{i=1}^t(1-{\beta }_i)$. The $\prod$ symbol is a repeated multiply (like $\Sigma$ but with $\times$ instead of $+$), and ${\beta }_i$ is how much noise we inject at each single step. See Denoising Diffusion Probabilistic Models (DDPM) — latent diffusion reuses this exact schedule.

• Why the topic needs it: this is the training data generator. To teach the network to remove noise, we must first add known amounts of it.

7. The noise predictor ${ϵ}_{\theta }$ and the U-Net

• Picture: a machine that looks at static-covered art and points at exactly which speckles are noise.
• Why a U-Net specifically? It sees the latent at many zoom levels at once (coarse shapes + fine texture), which is exactly what denoising needs. Details in U-Net Architecture.

8. The text hint $c$ and cross-attention

• Picture: each patch of the sketch raising its hand and asking the words "which of you describes me?"
• Why the topic needs it: without $c$, the U-Net denoises into some image; with $c$, it denoises into your image. This is how text controls the picture — see also Classifier-Free Guidance.

9. Sampling: turning noise back into a picture

• ${\sigma }_t$ (sigma) = how much fresh randomness we re-inject each step (controls variety). Setting it to zero gives the deterministic DDIM Sampling variant.
• Why loop instead of one jump? Each step is a small, easy correction; asking the network to leap from full static to a clean image in one go is too hard.

Prerequisite map

Equipment checklist

Test yourself — cover the right side and answer aloud.