Visual walkthrough — Latent diffusion (Stable Diffusion)

Step 1 — What is an image, really? (a grid of numbers)

WHAT. Before any maths, pin down the object we are manipulating. A colour image is a grid of tiny squares (pixels). Each pixel stores three numbers: how much Red, Green, Blue it glows. So a $512\times 512$ photo is a stack of three $512\times 512$ number-grids.

WHY. Everything downstream is "shuffling these numbers around." We must first count them, because the whole story is about that count being too big.

PICTURE. Look at the figure. The left board shows one grid, one pixel zoomed to reveal its three chalk-blue numbers $(R,G,B)$. Multiply it out:

Each factor is doing a plain job: rows count how tall, columns how wide, and the $3$ is "one number per colour." That product — call it the dimension of the image, meaning "how many independent knobs describe it" — is our enemy.

Step 2 — Why is that number our enemy? (the cost of denoising)

WHAT. Diffusion generates images by starting from pure static and cleaning it up over many steps with a network called a U-Net. The expensive part is the U-Net's self-attention, where every spatial location "talks to" every other location.

WHY this matters. "Everyone talks to everyone" means: if there are $N$ locations, there are $N\times N=N^2$ conversations. This is the single reason pixel-space diffusion is slow. We introduce the tool of counting pairs precisely because attention's cost is a count of pairs.

PICTURE. The figure draws $N=4$ dots and every arrow between them: $16$ arrows. Then $N=8$ dots: $64$ arrows. Double the dots, quadruple the work — that is what "$N^2$" looks like.

Here $N$ is the number of spatial squares, and squaring it counts the ordered pairs that attention must compute. For a $512\times 512$ grid, $N=262,144$, so $N^2$ is astronomical.

Step 3 — The escape: compress first, denoise later

WHAT. Instead of denoising the giant pixel grid, we first squeeze the image through an autoencoder into a much smaller grid called the latent $z$, denoise there, then expand back.

WHY a VAE and not a zip file? A zip file gives back exact bytes but its output is a jumble — you cannot smoothly slide between two zipped files. Diffusion needs to travel continuously from noisy to clean, so it needs a latent world where nearby points decode to nearby images. The VAE's extra training term (we meet it in Step 5) forces exactly that smoothness.

PICTURE. The figure is a funnel: the big pixel grid $x$ enters the encoder $\mathcal{E}$, becomes the tiny latent grid $z$, and the decoder $\mathcal{D}$ pushes $z$ back out to a reconstructed image $\hat{x}$.

• $x$ — the original image (dimension 786,432).
• $\mathcal{E}$ — the encoder network; read the script-E as "the squeezer."
• $z$ — the latent, the compressed grid we will actually work with.
• $\mathcal{D}$ — the decoder network, "the un-squeezer."
• $\hat{x}$ (x-with-a-hat) — the rebuilt image; the hat means "an estimate of," and $\approx$ means "roughly equals."

Step 4 — How big is the shrink? (the $f=8$ book-keeping)

WHAT. Stable Diffusion uses a downsampling factor $f=8$: each side of the grid is divided by $8$, and the compressed grid carries $c=4$ learned channels (not the original 3 colours).

WHY $f=8$ and $c=4$? These are the numbers that keep the picture perceptually intact while giving the biggest safe reduction. Let us just do the arithmetic so the payoff is concrete.

PICTURE. The figure shows the $512\times 512\times 3$ block shrinking to a $64\times 64\times 4$ block, with both totals chalked underneath.

Now compare the two worlds:

and — the part that really matters — the number of spatial locations drops from $N_{\text{pix}}=262,144$ to $N_{\text{lat}}=4096$, a factor of $64$. Feeding that into the $N^2$ rule from Step 2:

Each symbol: $N_{\text{pix}}$ = pixel-grid locations, $N_{\text{lat}}$ = latent-grid locations, the square is the "pairs" from Step 2, and the ratio is how many times cheaper the attention becomes.

Step 5 — Why the latent world is smooth (the KL guardrail)

WHAT. The VAE is trained with two pulls at once: (1) rebuild the image accurately, (2) keep every latent close to a standard bell-curve (a Gaussian centred at 0 with spread 1).

WHY. Pull (1) alone would let the encoder scatter latents anywhere it likes, leaving holes — regions that decode to garbage. Pull (2), written with the reconstruction-plus-KL recipe, herds all latents into one soft blob with no holes. We need the KL tool specifically because it measures how far a distribution is from the target bell-curve and can be minimised.

PICTURE. Left board: a plain autoencoder — latents in scattered islands, arrows into empty "holes" produce chalk-scribble garbage. Right board: the VAE — latents packed into one smooth cloud; every point in between decodes to something sensible.

• $\Vert x-\hat{x}{\Vert }^2$ — squared error, "how wrong is the rebuild" — small = faithful.
• $q(z\mid x)$ — the spread of latents the encoder assigns to image $x$.
• $p(z)$ — the target: a standard Gaussian, our "no-holes" reference.
• $D_{\mathrm{KL}}(\cdot \Vert \cdot )$ — the KL divergence, a number that is $0$ when the two distributions match and grows as they drift apart.

Step 6 — Diffusion inside the small world (add noise, learn to remove it)

WHAT. In the latent grid we corrupt the clean latent $z_0=\mathcal{E}(x)$ by mixing in random static, over $T$ tiny steps, until nothing remains but static. A U-Net then learns to predict the static so we can subtract it back off.

WHY predict the noise, not the image? The forward corruption is a clean linear blend, so the noise $ϵ$ is a fixed known target during training — easy to learn against. This is the same DDPM objective, just living in the 16k-dimensional latent instead of 786k-dimensional pixels.

PICTURE. A left-to-right filmstrip: $z_0$ (clear latent), progressively noisier $z_t$, ending at $z_T$ (pure static). Underneath, the exact mixing rule.

• $t$ — the step number, from $0$ (clean) up to $T$ (all static).
• ${\bar{\alpha }}_t$ ("alpha-bar") — a scheduled number sliding from near $1$ (early, mostly signal) to near $0$ (late, mostly noise).
• $\sqrt{{\bar{\alpha }}_t}{z}_0$ — the surviving fraction of the real latent.
• $\sqrt{1-{\bar{\alpha }}_t}ϵ$ — the injected fresh noise; the two square-roots are chosen so signal-power + noise-power stays balanced.
• $ϵ\sim \mathcal{N}(0,I)$ — standard Gaussian noise, one independent draw per latent number.

The U-Net ${ϵ}_{\theta }(z_t,t,c)$ is trained to output exactly that $ϵ$:

Here $\theta$ = the network's weights, $c$ = the text instruction (next step), and the squared bars again mean "how far off the guess is."

Step 7 — Steering with words (cross-attention to CLIP)

WHAT. So far denoising is blind — it would produce some plausible image. To obey "a corgi in a spacesuit," we feed the U-Net a text embedding $c$ from the CLIP encoder and let each latent square look up which words to obey via cross-attention.

WHY attention here? Different regions of the image need different words: the head must attend to "spacesuit," the body to "corgi." Attention is precisely the tool that lets each location choose what to listen to, with a weight for every word.

PICTURE. The latent grid on the left; the word tokens on the right. Chalk-pink arrows of varying thickness (attention weights) run from one grid square to the words it "reads." Thick arrow = strong influence.

• $Q$ (query) — comes from the image latent: "what am I, and what do I need?"
• $K$ (key) — comes from the text: "what does each word offer?"
• $V$ (value) — the text content actually pulled in once a match is found.
• $Q{K}^{\top }$ — every square-to-word match score.
• $\sqrt{d}$ — divides by the square-root of the feature size $d$ to keep the scores from blowing up.
• $\mathrm{softmax}$ — turns raw scores into positive weights summing to $1$ (a proper "how much to listen" mix).

To dial the obedience up, classifier-free guidance blends the with-text and without-text predictions — but that is a separate knob; here the machinery is the cross-attention itself.

Step 8 — Walking the noise back off (one reverse step)

WHAT. Generation runs the corruption backwards: from static $z_T$, at each step subtract the predicted noise and rescale, landing at the clean latent $z_0$.

WHY this exact formula? The forward mix in Step 6 is a known linear equation; algebraically inverting "signal + noise" for one step gives precisely the rule below. Each piece undoes one piece of the forward blend.

PICTURE. One reverse hop drawn as a vector diagram: start at $z_t$, the chalk-yellow arrow removes the predicted-noise component, the whole thing is rescaled, and a small chalk-blue wobble ${\sigma }_t\mathbf{z}$ adds diversity — arriving at $z_{t-1}$, a little cleaner.

• $\widehat{ϵ}={ϵ}_{\theta }(z_t,t,c)$ — the U-Net's noise guess for this step.
• ${\alpha }_t=1-{\beta }_t$ — the per-step signal-retention (single step, no bar).
• $\frac{1-{\alpha }_t}{\sqrt{1-{\bar{\alpha }}_t}}\widehat{ϵ}$ — exactly the slice of noise added at step $t$, peeled off.
• $\frac{1}{\sqrt{{\alpha }_t}}$ — restores the scale the forward step had shrunk.
• ${\sigma }_t\mathbf{z}$ — a fresh Gaussian nudge; set ${\sigma }_t=0$ and you get the deterministic DDIM variant.

Edge cases, walked explicitly:

• First step ($t=T$): $z_T$ is pure static, ${\bar{\alpha }}_T\approx 0$, so almost the whole bracket is "remove noise" — the model makes a bold first guess of the image's shape.
• Last step ($t=1$): we set ${\sigma }_1=0$ (no wobble at the very end) so the final $z_0$ is crisp, not re-noised.
• ${\sigma }_t\equiv 0$ everywhere: fully deterministic — same seed, same image (DDIM).
• Text dropped ($c=\varnothing$): the machinery still runs; it just produces some unconditioned image with no steering.

Finally, decode once: $x=\mathcal{D}(z_0)$ — the small clean latent is blown back up to the full $512\times 512$ picture.

The one-picture summary

Everything above is one loop. Text becomes a guide $c$; random static $z_T$ enters the small latent world; the U-Net (steered by $c$) removes noise over $T$ reverse steps; the decoder expands the clean $z_0$ back to pixels. All the heavy lifting happens in the $16,384$-number world, never the $786,432$-number one — that is the entire idea.