Latent diffusion (Stable Diffusion)

What Problem Does This Solve?

Problem with pixel-space diffusion: A 512×512 RGB image has 786,432 dimensions. Running U-Net denoising for many timesteps on this is computationally expensive, especially the self-attention layers.

Latent diffusion's solution:

Train a Variational Autoencoder (VAE) to compress images: $x \in \mathbb{R}^{H×W×3} \rightarrow z \in \mathbb{R}^{h×w×c}$ where $h,w \ll H,W$
Run diffusion only in latent space $z$
Decode final $z_0$ back to image with VAE decoder

Stable Diffusion uses $f=8$ downsampling: 512×512 → 64×64, with $c=4$ channels → 98% dimension reduction.

Architecture Components

1. The VAE (Autoencoder)

Why VAE and not simple autoencoder? VAE's KL term forces latent space to be continuous and smooth—nearby latents decode to similar images. Diffusion needs this smoothness to interpolate between noisy and clean latents.

Stable Diffusion specifics:

Downsampling factor $f=8$ : 512→64, 1024→128
Latent channels $c=4$ (learned, not RGB)
Trained separately on massive image datasets (LAION)
Trained with L2 pixel reconstruction loss + KL regularization (the released SD VAE uses these; note: no VGG/perceptual loss in the standard KL-autoencoder recipe)
Frozen during diffusion training—only used for encode/decode

2. The Diffusion U-Net (in Latent Space)

Instead of denoising pixels, we denoise latents $z_t$ :

Reverse process (denoising):

Why this works:

$z$ is 64×64×4 instead of 512×512×3 → U-Net operates on ~2% of original dimensions
Latents capture semantic structure, so U-Net learns "object shapes" not "pixel noise patterns"
Text conditioning $c$ (from CLIP) guides the denoising: "add fur texture" vs "add metal texture"

3. Text Conditioning (CLIP Embeddings)

Example: Prompt "a cat wearing sunglasses"

Latent region corresponding to head area attends to "cat" + "sunglasses" tokens
Latent region for body attends to "cat" token
U-Net uses this to inject appropriate features during denoising

Figure — Latent diffusion (Stable Diffusion)

Step-by-Step: How Stable Diffusion Generates an Image

Worked example Generation Pipeline

Input: Text prompt "a corgi in a spacesuit" Step 1: Encode text with CLIP $c = \text{CLIPEncoder}(\text{"a corgi in a spacesuit"})$

Step 2: Sample random latent noise $z_T \sim \mathcal{N}(0, I), \quad z_T \in \mathbb{R}^{64×64×4}$

Step 3: Denoise for $T$ steps (e.g., 50 steps) For $t = T, T-1, \ldots, 1$ :

\hat{\epsilon} &= \epsilon_\theta(z_t, t, c) \quad \text{(predict noise with U-Net)}\\ z_{t-1} &= \frac{1}{\sqrt{\alpha_t}}\left[\, z_t - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}}\,\hat{\epsilon} \,\right] + \sigma_t \mathbf{z} \end{aligned}$$ where $\mathbf{z} \sim \mathcal{N}(0,I)$ is new noise (DDPM sampling), $\sigma_t$ controls stochasticity, and $\alpha_t = 1-\beta_t$. **Why this step?** We're reversing the noise addition. The $\frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}}\hat{\epsilon}$ term **removes the predicted noise component**, the leading $\frac{1}{\sqrt{\alpha_t}}$ factor rescales the whole bracket, and $\sigma_t\mathbf{z}$ adds controlled randomness for diversity. **Step 4:** Decode latent to image $$x = \mathcal{D}(z_0) \quad \text{(VAE decoder)}$$ **Output:** 512×512 RGB image of corgi in spacesuit.

Why Latent Space? A Derivation from First Principles

Question: Why is denoising in latent space more efficient than pixel space?

Computational cost per denoising step (schematically): $\text{cost} \propto (\text{number of spatial locations}) × (\text{channels}) × (\text{per-token work})$ and the self-attention part specifically scales quadratically in the number of spatial locations $N = h\cdot w$ : $\text{attention cost} \propto N^2 = (h w)^2$

For pixel-space (DDPM):

Spatial locations: $512×512 = 262{,}144$
Channels: 3 (RGB)
Total dim: $262{,}144 × 3 = 786{,}432$

For latent-space (Stable Diffusion):

Spatial locations: $64×64 = 4{,}096$
Channels: 4
Total dim: $4{,}096 × 4 = 16{,}384$

Dimension ratio: $\frac{786{,}432}{16{,}384} = 48×$ fewer dimensions.

Since the attention cost scales as $N^2$ , the reduction in spatial locations from $262{,}144$ to $4{,}096$ (a $64×$ drop in $N$ ) yields an attention-cost reduction of roughly: $\frac{N_\text{pix}^2}{N_\text{lat}^2} = \left(\frac{262{,}144}{4{,}096}\right)^2 = 64^2 = 4096×$ while element-wise convolutional work scales linearly with $N$ (~ $64×$ cheaper). The practical end-to-end speedup depends on how attention-heavy the U-Net is; empirically it is large (roughly an order of magnitude or more), which is exactly why latent diffusion made high-res text-to-image feasible on consumer GPUs.

But why no quality loss?

The VAE is trained to reconstruct images (L2 loss) with KL regularization. It discards imperceptible high-frequency detail and spatial redundancy (neighboring pixels being nearly identical) while preserving semantic content—object shapes, colors, composition. For generative tasks we care about semantic correctness ("does it look like a cat?"), not exact pixel values, so this compression is essentially perceptually harmless.

Common mistake Common Misconception: "Latent diffusion is lower quality"

Wrong idea: "Compressing to 64×64 must lose detail—pixel diffusion should be sharper."

Why it feels right: Compression = information loss, right? And 98% dimension reduction sounds extreme.

The fix: The VAE is lossy but perceptually near-lossless. It discards imperceptible high-frequency noise and spatial redundancy. The 64×64×4 latent still encodes all semantic content—object shapes, textures, colors, composition.

Why it holds up: Latent diffusion matches or beats pixel-space DDPM at high resolution in practice, because:

Training on latents is far cheaper → more training iterations → better convergence
Latent space has better inductive bias (semantic structure, not pixel noise)
Compute budget goes to larger models and datasets instead of fighting dimensionality

Common mistake Another Mistake: "I can just use any autoencoder"

Wrong idea: "I'll train a simple convolutional autoencoder, then do diffusion in its latent space."

Why it fails: Non-regularized autoencoders create discontinuous latent spaces with "holes"—regions that decode to garbage. If your diffusion process passes through a hole during denoising, the image breaks.

Why the KL term matters: The KL divergence $D_{KL}(q(z|x) \| p(z))$ pushes the latent distribution toward a smooth, Gaussian-like prior. This means:

Interpolating between two latents gives sensible images (no holes)
Adding noise (diffusion forward process) stays in valid latent regions
Denoising (reverse process) won't "fall off" the manifold

Steel-man the mistake: You might think "KL divergence is just a regularizer, I can skip it for speed." But diffusion requires traversing the latent space continuously—the smoothness is not optional.

Key Hyperparameters and Design Choices

Parameter	Stable Diffusion v1.5	Why This Value?
Latent channels $c$	4	Balance: 3 is undercomplete (loses info), 8 is redundant. 4 captures structure well.
Downsampling $f$	8	$f=4$ → still large latents (slow), $f=16$ → too lossy (VAE artifacts). 8 is sweet spot.
U-Net timesteps $T$	1000 (training)	Standard DDPM schedule. Higher $T$ = smoother noise schedule but slower training.
Sampling steps	20–50 (inference)	DDIM/other samplers skip steps. ~50 steps ≈ near-max quality (diminishing returns beyond).
CFG scale	7.5	Classifier-Free Guidance scale. Higher = more prompt adherence, lower = more diversity.

Classifier-Free Guidance (CFG): During sampling, we interpolate between conditional and unconditional predictions: $\hat{\epsilon} = \epsilon_\theta(z_t, t, \varnothing) + s \cdot (\epsilon_\theta(z_t, t, c) - \epsilon_\theta(z_t, t, \varnothing))$ where $s$ is guidance scale (7.5), $\varnothing$ is null prompt. This amplifies the text conditioning.

Worked example Detailed Worked Example: Why Latent Is Cheaper

Setup: Generate a 512×512 image.

Pixel-space diffusion (DDPM):

Number of spatial tokens: $N_\text{pix} = 512×512 = 262{,}144$
Self-attention over these tokens costs $\propto N_\text{pix}^2 \approx 6.9×10^{10}$ (relative units)

Latent diffusion (Stable Diffusion):

Encode once: $\mathcal{E}(x)$ maps 512→64
Number of spatial tokens: $N_\text{lat} = 64×64 = 4{,}096$
Self-attention costs $\propto N_\text{lat}^2 \approx 1.7×10^{7}$ (relative units)
Decode once: $\mathcal{D}(z_0)$ maps 64→512

Attention-cost ratio: $\dfrac{N_\text{pix}^2}{N_\text{lat}^2} = \left(\dfrac{262{,}144}{4{,}096}\right)^2 = 64^2 = 4096×$ cheaper in the attention layers.

Why this step? Attention compares every token with every other token, so halving/eighth-ing the spatial resolution multiplies down quadratically. The convolutional layers scale linearly (~ $64×$ ), and VAE encode/decode happen only once (not per step), so their cost is negligible over 20–50 denoising steps. Net effect: latent diffusion makes high-resolution generation practical on consumer GPUs (seconds, not the impractically long times pixel-space attention would require at 512²).

Recall Explain to a 12-year-old

Imagine you want to draw a super detailed picture, but you only have a tiny notepad. Drawing all the details directly would take forever!

Here's the trick Stable Diffusion uses:

Shrink the picture – Take your big picture and compress it into a tiny "summary" (like shrinking a photo on your phone). This summary keeps all the important stuff—the cat's face, the colors—but throws away tiny details like individual hairs.
Draw in the tiny space – Now, instead of drawing on a huge canvas, you draw on this tiny summary. It's way faster! You can sketch the cat, add sunglasses, fix mistakes—all super quick.
Blow it back up – When you're done, you "un-compress" the tiny summary back to a big, detailed picture. Magic!

Why does this work? Because most of a picture is repetitive—blue sky pixels are all similar, grass is just repeated texture. The "summary" keeps the unique parts (cat's eyes, sunglasses shape) and lets you recreate the repetitive parts later. You're working on the meaning of the picture, not every single pixel.

Connections

Variational Autoencoders (VAE) – the compression backbone
Denoising Diffusion Probabilistic Models (DDPM) – latent diffusion applies DDPM's noise schedule in latent space
U-Net Architecture – the denoising model (with attention layers for conditioning)
CLIP Text Encoder – converts prompts to embeddings for cross-attention
Classifier-Free Guidance – technique to strengthen text conditioning (that CFG scale slider)
DDIM Sampling – faster sampling algorithm (why 20–50 steps, not 1000)
Attention Mechanisms – how text conditions the image generation
Perceptual Loss – related idea; note the SD KL-autoencoder uses L2+KL, not VGG

#flashcards/ai-ml

What is the key innovation of latent diffusion models? :: Running the diffusion process (noise addition and denoising) in a low-dimensional latent space produced by a VAE, rather than directly in pixel space, achieving a large speedup with negligible quality loss.

Why use a VAE (with KL) instead of a plain autoencoder for latent diffusion?

The KL term pushes the latent distribution toward a smooth Gaussian-like prior, preventing "holes" where interpolation breaks. Diffusion requires traversing latent space continuously—without smoothness, denoising produces artifacts.

What is the dimension reduction in Stable Diffusion v1.5?

512×512×3 (786,432 dimensions) → 64×64×4 (16,384 dimensions), a ~98% reduction via f=8 downsampling VAE.

What is the latent diffusion forward process equation?

z_t = \sqrt{\bar{\alpha}_t} z_0 + \sqrt{1-\bar{\alpha}_t}\epsilon

where

z_0 = \mathcal{E}(x)

is the encoded image and

\epsilon \sim \mathcal{N}(0,I)

is noise.

What is the training objective for latent diffusion? :: $\mathcal{L} = \mathbb{E}_{z_0, t, \epsilon, c}[\|\epsilon - \epsilon_\theta(z_t, t, c)\|^2]$ — train U-Net to predict noise $\epsilon$ given noisy latent $z_t$ , timestep $t$ , and text conditioning $c$ .

What is the DDPM reverse-step update?

z_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left[z_t - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}}\hat{\epsilon}\right] + \sigma_t \mathbf{z}

, with

\alpha_t=1-\beta_t

and

\mathbf{z}\sim\mathcal{N}(0,I)

How does text conditioning work in Stable Diffusion?

Text is encoded by CLIP into embedding

c

, then injected via cross-attention in the U-Net. Query comes from image features, key/value from text, letting each spatial region attend to relevant prompt tokens.

What loss trains the Stable Diffusion VAE?

Pixel-level L2 reconstruction loss + a KL divergence regularization term (the standard KL-autoencoder recipe); no VGG/perceptual loss in that recipe.

Why does self-attention make pixel-space diffusion so expensive?

Attention cost scales as

N^2

in the number of spatial tokens

N=hw

. At 512² there are 262,144 tokens; at 64² only 4,096, so attention cost drops by

(262144/4096)^2 = 4096\times

What are the four Stable Diffusion generation steps?

1) Encode text with CLIP →

c

, 2) Sample noise

z_T \sim \mathcal{N}(0,I)

, 3) Denoise for T steps using U-Net

\epsilon_\theta(z_t, t, c)

, 4) Decode

z_0

to image with VAE decoder

\mathcal{D}(z_0)

What is Classifier-Free Guidance (CFG)?

\hat{\epsilon} = \epsilon_\theta(z_t, t, \varnothing) + s(\epsilon_\theta(z_t, t, c) - \epsilon_\theta(z_t, t, \varnothing))

. Scale

s

controls prompt adherence vs diversity.

Why can't diffusion "fall off" the latent manifold in the SD VAE space?

KL regularization makes latents near-Gaussian and smooth, so all points interpolate to valid images—no "holes" to fall into during denoising.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ek image ko denoise karna directly 512×512×3 dimensions me matlab lagbhag 7.8 lakh numbers ke saath khelna—yeh bahut hi expensive aur slow hota hai, khaaskar jab diffusion me hume kai timesteps ke liye U-Net chalana padta hai. Latent Diffusion ka core idea yeh hai ki natural images me bahut saara redundancy hota hai—paas ke pixels aksar similar hote hain, textures repeat hote hain. Toh pehle hum ek autoencoder (VAE) se image ko ek chhote se latent space me compress kar dete hain, jaise 64×64×4 (matlab sirf ~2% dimensions bache), fir denoising wahi latent space me karte hain, aur end me VAE decoder se wapas pixels me convert kar dete hain. Yeh Stable Diffusion ka pura backbone hai.

Ab yeh kyun kaam karta hai? Kyunki VAE image ka sirf semantic content rakhta hai aur faltu redundancy phenk deta hai. Iska ek khaas point hai ki hum VAE use karte hain simple autoencoder nahi—VAE ka KL divergence term latent space ko smooth aur continuous banata hai, jisse paas ke latents similar images me decode hote hain. Diffusion ko yahi smoothness chahiye taaki noisy se clean latent ke beech transition sahi se ho paaye. Aur text conditioning ke liye CLIP embeddings use hote hain jo cross-attention ke through U-Net ko batate hain ki kaha "cat" banana hai aur kaha "sunglasses"—har spatial region apne relevant text tokens ko dhoondh leta hai.

Yeh matter isliye karta hai kyunki isi trick ki wajah se aaj tum apne normal GPU par hi high-quality image generate kar paate ho—warna pixel-space diffusion itna heavy hai ki common hardware par chalana mushkil ho jaata. Compute aur memory dono bachate hain, aur upar se latents par denoising karne se model asli me "object shapes" seekhta hai, sirf "pixel noise patterns" nahi. Isliye Stable Diffusion type models itne practical aur popular bane—yeh compression + diffusion + conditioning ka smart combination hai jo quality aur efficiency dono deta hai.

What Problem Does This Solve?

Architecture Components

1. The VAE (Autoencoder)

2. The Diffusion U-Net (in Latent Space)

3. Text Conditioning (CLIP Embeddings)

Step-by-Step: How Stable Diffusion Generates an Image

Why Latent Space? A Derivation from First Principles

Key Hyperparameters and Design Choices

Connections

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Generative Models

Connections