Latent diffusion (Stable Diffusion): Exercises

Before we compute anything, let us pin down the numbers that appear again and again, so no symbol is ever used unexplained.

Definition The quantities we will reuse

$H \times W$ — the image's height and width in pixels. A picture is a grid; $H$ counts rows, $W$ counts columns.
$f$ — the downsampling factor of the VAE encoder. " $f = 8$ " means each side of the image shrinks by $8$ : a $512$ -pixel side becomes $512/8 = 64$ .
$h = H / f, w = W / f$ — the latent grid height and width after compression.
$C$ — the number of latent channels. Just as an RGB pixel has $3$ numbers, a latent cell here has $C = 4$ learned numbers. (We use a capital $C$ for channels so it is never confused with the text context $c$ below.)
$N = h \cdot w$ — the number of spatial locations (cells) the U-Net's attention has to compare against each other.
$T$ — the total number of diffusion steps in the schedule. $t$ is a single timestep index running $t = 0, 1, \dots, T$ ; $t = 0$ is the clean latent, $t = T$ is (almost) pure noise. Sampling walks $t$ downward from $T$ to $0$ .
$z_{0}$ — the clean latent code, $z_{0} = E (x)$ : what you get by encoding a real image. $z_{t}$ is that same latent after $t$ steps of noise have been mixed in.
$ϵ$ — standard Gaussian noise, $ϵ \sim N (0, I)$ : a tensor the same shape as $z_{0}$ , every entry an independent random number of mean $0$ and variance $1$ .
$c$ — the text-conditioning context: the embedding $c \in R^{77 \times 768}$ produced by the CLIP Text Encoder from the prompt. It is not a channel count; it is the "what to draw" signal fed into the U-Net through cross-attention.
$\overset{α}{ˉ}_{t} \in [0, 1]$ — the cumulative signal-survival fraction at step $t$ . At $t = 0$ it is $\approx 1$ (all clean signal), and it shrinks toward $0$ as noise accumulates. It appears in the forward process $z_{t} = \overset{α}{ˉ}_{t} z_{0} + 1 - \overset{α}{ˉ}_{t} ϵ,$ which mixes the clean latent $z_{0}$ with the Gaussian noise $ϵ$ in amounts set by $\overset{α}{ˉ}_{t}$ .

The figure above is the mental picture behind every problem: a big pixel grid squeezed into a tiny latent grid, denoised there, then decoded back.

Stable Diffusion uses $f = 8$ and a $512 \times 512$ input image. What is the latent spatial grid size $h \times w$ , and what is the total number of latent scalars (including channels, $C = 4$ )?

Recall Solution

What we do: divide each spatial side by $f$ , then multiply by channels. $h = w = \frac{512}{8} = 64 \Rightarrow h \times w = 64 \times 64.$ Total latent scalars: $N \cdot C = (64 \cdot 64) \cdot 4 = 4096 \cdot 4 = 16, 384.$ Answer: latent grid $64 \times 64$ , total $16, 384$ numbers.

Why dividing by $f$ is the right move (intuition): the encoder is a stack of stride- $2$ convolutions; each halves the side length. Three such halvings give $2^{3} = 8 = f$ , so one latent cell summarises an $8 \times 8$ patch of pixels. We shrink by $f$ per side precisely because neighbouring pixels are highly redundant — an $8 \times 8$ block of a smooth region carries almost the same information as a single well-chosen number, so throwing away that redundancy costs little while cutting the grid dramatically.

Exercise 1.2

Which component is frozen during diffusion training, and which is being trained? Name both.

Recall Solution

Frozen: the VAE encoder $E$ and decoder $D$ . They were pretrained separately and are only called to encode/decode.
Trained: the diffusion U-Net $ϵ_{θ} (z_{t}, t, c)$ , which learns to predict the added noise $ϵ$ inside latent space (here $c$ is the text context, not a channel count). The CLIP Text Encoder is also typically frozen (used only to produce $c$ ).

Why freeze the VAE (intuition): the U-Net's whole job is to learn the distribution of clean latents $z_{0} = E (x)$ . If $E$ kept changing, that distribution would move — you cannot learn to hit a target that keeps sliding. Freezing fixes the coordinate system so denoising has a stable meaning.

Level 2 — Application

Exercise 2.1

A $1024 \times 1024$ image is encoded with $f = 8$ , $C = 4$ . Compute the pixel dimension, the latent dimension, and the dimension-reduction ratio.

Recall Solution

Pixel dimension (RGB, so $3$ channels): $1024 \cdot 1024 \cdot 3 = 3, 145, 728.$ Latent dimension: side $= 1024/8 = 128$ , so $128 \cdot 128 \cdot 4 = 65, 536.$ Ratio: $\frac{3 , 145 , 728}{65 , 536} = 48.$ Answer: $48 \times$ fewer dimensions — the same ratio as the $512$ case, because the ratio only depends on $f$ and channel counts, not on image size: $\frac{H W \cdot 3}{( H / f ) ( W / f ) \cdot C} = \frac{3 f ^{2}}{C} = \frac{3 \cdot 64}{4} = 48.$

Exercise 2.2

Given the forward process $z_{t} = \overset{α}{ˉ}_{t} z_{0} + 1 - \overset{α}{ˉ}_{t} ϵ$ (recall $z_{0}$ is the clean latent code and $ϵ \sim N (0, I)$ is standard Gaussian noise) with $\overset{α}{ˉ}_{t} = 0.36$ , what fraction of the latent's standard-deviation budget comes from the clean signal, and what fraction from noise? (Assume $z_{0}$ and $ϵ$ each have unit variance.)

Recall Solution

What we do: the coefficients in front of $z_{0}$ and $ϵ$ are the amplitude weights.

Signal amplitude: $\overset{α}{ˉ}_{t} = 0.36 = 0.6$ .
Noise amplitude: $1 - \overset{α}{ˉ}_{t} = 0.64 = 0.8$ .

Why these add in quadrature: variances of independent parts add, so the total variance is $0. 6^{2} + 0. 8^{2} = 0.36 + 0.64 = 1$ . The variance is fully accounted for: $36%$ from signal, $64%$ from noise. Answer: signal amplitude $0.6$ , noise amplitude $0.8$ ; in variance terms $36%$ signal / $64%$ noise.

Level 3 — Analysis

Exercise 3.1

Self-attention cost scales as $N^{2}$ where $N = h \cdot w$ . Compare pixel-space ( $512 \times 512$ , $N_{pix} = 262, 144$ ) with latent-space ( $64 \times 64$ , $N_{lat} = 4, 096$ ). By what factor does the attention cost shrink? By what factor does the linear (convolution) work shrink?

Recall Solution

Linear part (scales $\propto N$ ): $\frac{N _{pix}}{N _{lat}} = \frac{262 , 144}{4 , 096} = 64 \times .$ Attention part (scales $\propto N^{2}$ ): $\frac{N _{pix}^{2}}{N _{lat}^{2}} = 6 4^{2} = 4096 \times .$ Answer: $64 \times$ cheaper convolution, $4096 \times$ cheaper self-attention. This is exactly why latent diffusion made high-res generation feasible: the quadratic term is where pixel-space drowns.

The plot below makes the gap visual: the two operating points sit on the two curves — the latent point drops off the steep quadratic curve entirely, while the pixel point sits high up on it. Look at how far apart the magenta ( $N^{2}$ ) and orange ( $N$ ) curves are at the pixel point versus how close they are at the latent point.

Exercise 3.2

The CFG update combines a conditional and an unconditional noise prediction: $\overset{ϵ}{^} = ϵ_{θ} (z_{t}, t, \emptyset) + s (ϵ_{θ} (z_{t}, t, c) - ϵ_{θ} (z_{t}, t, \emptyset)) .$ Here $c$ is the text context, $\emptyset$ means "empty prompt" and $s$ is the guidance scale. If the unconditional prediction is $ϵ_{u} = 0.20$ and the conditional is $ϵ_{c} = 0.50$ (one representative component), compute $\overset{ϵ}{^}$ for $s = 1$ and $s = 7.5$ .

Recall Solution

What we do: substitute into the formula. $s = 1$ recovers the plain conditional prediction.

$s = 1$ : $\overset{ϵ}{^} = 0.20 + 1 \cdot (0.50 - 0.20) = 0.20 + 0.30 = 0.50.$
$s = 7.5$ : $\overset{ϵ}{^} = 0.20 + 7.5 \cdot (0.30) = 0.20 + 2.25 = 2.45.$

What it means: higher $s$ pushes the prediction further along the "text direction" $(ϵ_{c} - ϵ_{u})$ , amplifying how strongly the prompt steers the image — at the cost of saturation/artifacts if $s$ is too large. Answer: $0.50$ and $2.45$ .

Level 4 — Synthesis

Exercise 4.1

You run DDIM Sampling with $50$ steps on a $512 \times 512$ image. Each U-Net call is one "forward pass". Using CFG, each step needs two forward passes (conditional + unconditional). How many U-Net forward passes does generating one image require? How many VAE decoder calls?

Recall Solution

U-Net passes: $50 steps \times 2 = 100$ forward passes. VAE decoder calls: the decoder $D$ is called once, only at the end on $z_{0}$ , to turn the final latent into pixels. Answer: $100$ U-Net passes, $1$ decoder call. (The encoder $E$ is called $0$ times in pure text-to-image — we start from pure noise $z_{T}$ , not from an image.)

Exercise 4.2

Design check: you want to generate a $768 \times 768$ image with $f = 8$ , $C = 4$ . (a) What is the latent grid $N$ ? (b) Relative to the $512$ case ( $N = 4096$ ), by what factor does self-attention cost grow?

Recall Solution

(a) side $= 768/8 = 96$ , so $N = 96 \cdot 96 = 9216$ . (b) attention $\propto N^{2}$ : $\frac{921 6 ^{2}}{409 6 ^{2}} = (\frac{9216}{4096})^{2} = (\frac{9}{4})^{2} = \frac{81}{16} = 5.0625 \times .$ Answer: $N = 9216$ ; self-attention becomes $\approx 5.06 \times$ more expensive than at $512$ . Doubling image area more than doubles attention cost — the quadratic bites even inside latent space.

Level 5 — Mastery

Exercise 5.1

Full pipeline trace. A user generates one $512 \times 512$ image, prompt "a corgi in a spacesuit", $T = 50$ DDIM steps, CFG scale $s = 7.5$ , batch size $1$ . List in order every distinct tensor shape the data passes through, from prompt to pixels, and state which module produces each.

Recall Solution

Stage	Module	Output shape
1. Tokenise + encode text	CLIP Text Encoder (frozen)	$c \in R^{77 \times 768}$
2. Sample start noise	—	$z_{T} \in R^{64 \times 64 \times 4}$
3. Each denoise step ( $\times 50$ , $\times 2$ CFG)	U-Net $ϵ_{θ}$	$\overset{ϵ}{^} \in R^{64 \times 64 \times 4}$
4. Update latent	DDIM rule	$z_{t - 1} \in R^{64 \times 64 \times 4}$
5. Final latent	—	$z_{0} \in R^{64 \times 64 \times 4}$
6. Decode	[[Variational Autoencoders (VAE)	VAE decoder $D$ ]] (frozen)

Key structural fact: the entire loop (steps 2–5) lives in the $64 \times 64 \times 4$ latent grid — the decoder is touched exactly once. The text context $c$ (shape $77 \times 768$ ) enters through cross-attention: $Q$ from the latent, $K, V$ from $c$ .

Exercise 5.2

Sanity-check the noise budget across a whole schedule. Suppose $\overset{α}{ˉ}_{t}$ decays geometrically as $\overset{α}{ˉ}_{t} = (0.98)^{t}$ over timesteps $t$ . Find $t$ (rounded to the nearest integer) at which the latent is "half signal by variance", i.e. $\overset{α}{ˉ}_{t} = 0.5$ .

Recall Solution

What we do: solve $(0.98)^{t} = 0.5$ using logarithms — the natural tool for "how many multiplications of $0.98$ reach $0.5$ ?" $t = \frac{l n 0.5}{l n 0.98} = \frac{- 0.693147}{- 0.020203} \approx 34.31.$ Rounded: $t = 34$ . Check: $(0.98)^{34} = 0.5033 \approx 0.5$ . ✓ Meaning: by step $34$ of this schedule, signal and noise contribute roughly equal variance — the midpoint of the corruption process.

Recall Rapid self-check (cloze)

The VAE downsampling factor in Stable Diffusion is == $f = 8$ , giving a $64 \times 64$ == latent grid from a $512$ image. Self-attention cost scales as == $N^{2}$ == in the number of spatial locations $N = h \cdot w$ . Under CFG, each sampling step requires two U-Net forward passes. How many VAE decoder calls per generated image? ::: Exactly one, on the final latent $z_{0}$ . Why do variances (not amplitudes) of signal and noise add to 1? ::: Because $z_{0}$ and $ϵ$ are independent, so $(\overset{α}{ˉ}_{t})^{2} + (1 - \overset{α}{ˉ}_{t})^{2} = 1$ . What does the symbol $c$ denote (and what does it NOT denote)? ::: The text-conditioning context from CLIP ( $77 \times 768$ ); it is not the channel count $C$ .

Exercises — Latent diffusion (Stable Diffusion)

Level 1 — Recognition

Exercise 1.1

Exercise 1.2

Level 2 — Application

Exercise 2.1

Exercise 2.2

Level 3 — Analysis

Exercise 3.1

Exercise 3.2

Level 4 — Synthesis

Exercise 4.1

Exercise 4.2

Level 5 — Mastery

Exercise 5.1

Exercise 5.2