Text-to-image conditioning (CLIP): Exercises — AI-ML deep dive

Before we start, one symbol reminder so nothing is used before it is named:

Here $a \cdot b = \sum_{k} a_{k} b_{k}$ (multiply matching entries, add them up) and $∥ a ∥ = a \cdot a$ (the arrow's length). We use cosine and not raw dot product because we only care about which way an embedding points (its meaning), not how long it is — see contrastive learning.

Figure — Text-to-image conditioning (CLIP)

Level 1 — Recognition

Exercise 1.1

Two unit-length embeddings point in exactly the same direction. What is their cosine similarity $s$ ? What if they are perpendicular? What if they point in opposite directions?

Recall Solution

Cosine similarity is $cos θ$ where $θ$ is the angle between the arrows.

Same direction: $θ = 0^{\circ}$ , so $s = cos 0^{\circ} = 1$ .
Perpendicular: $θ = 9 0^{\circ}$ , so $s = cos 9 0^{\circ} = 0$ .
Opposite: $θ = 18 0^{\circ}$ , so $s = cos 18 0^{\circ} = - 1$ .

This is exactly why $s \in [- 1, 1]$ : $1$ = "same meaning", $0$ = "unrelated", $- 1$ = "opposite meaning".

Exercise 1.2

In the CLIP similarity matrix $S$ for a batch, which entries are the positives (should be large) and which are the negatives (should be small)?

Recall Solution

The diagonal entries $s_{ii}$ are positives — image $i$ paired with its own caption $i$ . Every off-diagonal entry $s_{ij}$ with $i \neq = j$ is a negative — image $i$ against someone else's caption. Training pushes the diagonal up and everything off-diagonal down.

Level 2 — Application

Exercise 2.1

Let $v_{I} = (3, 4)$ and $v_{T} = (4, 3)$ . Compute the cosine similarity $s$ .

Recall Solution

Dot product: $v_{I} \cdot v_{T} = 3 \cdot 4 + 4 \cdot 3 = 12 + 12 = 24$ . Lengths: $∥ v_{I} ∥ = 3^{2} + 4^{2} = 25 = 5$ and likewise $∥ v_{T} ∥ = 5$ . $s = \frac{24}{5 \cdot 5} = \frac{24}{25} = 0.96.$ The arrows nearly overlap (angle $\approx 16. 3^{\circ}$ ), so $s$ is close to $1$ .

Exercise 2.2

A batch of 3 has diagonal similarities and, for image 1, the row $s_{1 \cdot} = (0.9, 0.1, 0.2)$ . With temperature $τ = 0.1$ , compute $p (1 ∣ 1)$ — the softmax probability that image 1 matches text 1.

Recall Solution

Divide by $τ$ : logits $= (9, 1, 2)$ . Exponentiate: $e^{9} \approx 8103.08$ , $e^{1} \approx 2.71828$ , $e^{2} \approx 7.38906$ . $p (1 ∣ 1) = \frac{e ^{9}}{e ^{9} + e ^{1} + e ^{2}} = \frac{8103.08}{8113.19} \approx 0.99875.$ The small $τ$ made the correct match dominate — the softmax is very confident.

Exercise 2.3

Same row $s_{1 \cdot} = (0.9, 0.1, 0.2)$ but now $τ = 1$ . Recompute $p (1 ∣ 1)$ and comment.

Recall Solution

Logits $= (0.9, 0.1, 0.2)$ . Exponentials: $e^{0.9} \approx 2.4596$ , $e^{0.1} \approx 1.1052$ , $e^{0.2} \approx 1.2214$ . $p (1 ∣ 1) = \frac{2.4596}{2.4596 + 1.1052 + 1.2214} = \frac{2.4596}{4.7862} \approx 0.5139.$ With a large temperature the distribution is much flatter — only $\approx 51%$ confidence even though $s_{11}$ is the biggest. This is why CLIP uses a small $τ$ : it sharpens the softmax so correct pairs are strongly rewarded.

Level 3 — Analysis

Exercise 3.1

Show that as $τ \to 0^{+}$ (and assuming $s_{11}$ is strictly the largest entry in row 1), $p (1 ∣ 1) \to 1$ . Explain in words what this means for training.

Recall Solution

Write $p (1 ∣ 1) = \frac{e ^{s_{11} / τ}}{\sum _{j} e ^{s_{1 j} / τ}} = \frac{1}{\sum _{j} e ^{(s_{1 j} - s_{11}) / τ}}$ (dividing top and bottom by $e^{s_{11} / τ}$ ). For the correct term $j = 1$ : the exponent is $0$ , giving $1$ . For every wrong $j$ : $s_{1 j} - s_{11} < 0$ , so $(s_{1 j} - s_{11}) / τ \to - \infty$ as $τ \to 0^{+}$ , hence $e^{(s_{1 j} - s_{11}) / τ} \to 0$ . Therefore the denominator $\to 1$ and $p (1 ∣ 1) \to 1$ . Meaning: a tiny temperature turns the softmax into an (almost) hard argmax. Great for confident classification, but too small a $τ$ makes gradients vanish for near-ties — the model stops learning fine distinctions. That is why $τ$ is learned, not fixed at $0$ .

Exercise 3.2

The CLIP loss is symmetric: $L_{CLIP} = \frac{1}{2} (L_{I \to T} + L_{T \to I})$ . Give a concrete failure that could occur if you dropped one direction and only trained $L_{I \to T}$ .

Recall Solution

$L_{I \to T}$ only normalises across columns for a fixed image — "given this image, pick the right caption". Nothing in it forces "given this caption, pick the right image". The text encoder could collapse many captions onto nearly the same vector: as long as each image still finds its caption slightly more than the others in its own row, $L_{I \to T}$ can be low. The reverse loss $L_{T \to I}$ normalises across rows for a fixed text, punishing exactly that collapse. Training both directions makes both encoders learn discriminative representations.

Exercise 3.3

In zero-shot classification (Example 3 of the parent), why does the prompt template "a photo of a dog" typically beat the bare word "dog"?

Recall Solution

CLIP was trained on real internet captions, which are almost always full phrases ("a photo of ...", "an image of ..."). The text encoder therefore lives in a region of embedding space shaped by such phrasing. Feeding a bare token "dog" lands in a slightly off-distribution spot, so its embedding is a poorer probe. Matching the training distribution with a template moves the text embedding to where the image embeddings expect their matches, raising the correct-class similarity. This is a form of transfer prompt-engineering.

Level 4 — Synthesis

Exercise 4.1

Classifier-free guidance produces $\tilde{ϵ} = ϵ_{\emptyset} + w (ϵ_{c} - ϵ_{\emptyset})$ , where $ϵ_{c} = ϵ_{θ} (z_{t}, t, c)$ and $ϵ_{\emptyset} = ϵ_{θ} (z_{t}, t, \emptyset)$ . Show algebraically that this equals $w ϵ_{c} + (1 - w) ϵ_{\emptyset}$ , and interpret the special cases $w = 0$ and $w = 1$ .

Recall Solution

Expand: $ϵ_{\emptyset} + w ϵ_{c} - w ϵ_{\emptyset} = w ϵ_{c} + (1 - w) ϵ_{\emptyset}$ . ✓

$w = 0$ : $\tilde{ϵ} = ϵ_{\emptyset}$ — pure unconditional generation; the prompt is ignored, output is random-but-plausible images.
$w = 1$ : $\tilde{ϵ} = ϵ_{c}$ — the plain conditional prediction, no amplification. Faithful but low-diversity clamping only begins for $w > 1$ . See classifier-free guidance for the full picture.

Exercise 4.2

Suppose at some pixel $ϵ_{\emptyset} = 0.2$ and $ϵ_{c} = 0.5$ . Compute the guided prediction $\tilde{ϵ}$ for guidance scales $w = 1, 7.5, 15$ .

Recall Solution

Use $\tilde{ϵ} = ϵ_{\emptyset} + w (ϵ_{c} - ϵ_{\emptyset})$ with $ϵ_{c} - ϵ_{\emptyset} = 0.3$ .

$w = 1$ : $0.2 + 1 (0.3) = 0.5$ .
$w = 7.5$ : $0.2 + 7.5 (0.3) = 0.2 + 2.25 = 2.45$ .
$w = 15$ : $0.2 + 15 (0.3) = 0.2 + 4.5 = 4.7$ . Notice the guided value shoots far past the conditional $0.5$ : guidance extrapolates along the prompt direction, which sharpens prompt-fidelity but, if $w$ is huge, over-saturates the image. This is the geometric picture in the figure below.

Exercise 4.3

The guidance term $(ϵ_{c} - ϵ_{\emptyset})$ is claimed to approximate $\nabla_{z_{t}} lo g p (c ∣ z_{t})$ (the classifier gradient). Using $\nabla_{z_{t}} lo g p (z_{t} ∣ c) = \nabla_{z_{t}} lo g p (z_{t}) + \nabla_{z_{t}} lo g p (c ∣ z_{t})$ , explain which learned network plays the role of each term.

Recall Solution

A diffusion model's noise prediction is proportional to a score (negative gradient of a log-density): $ϵ_{θ} (z_{t}, t, c) \propto - \nabla_{z_{t}} lo g p (z_{t} ∣ c)$ and $ϵ_{θ} (z_{t}, t, \emptyset) \propto - \nabla_{z_{t}} lo g p (z_{t})$ .

$ϵ_{\emptyset}$ learns the marginal score $\nabla_{z_{t}} lo g p (z_{t})$ (images in general).
$ϵ_{c}$ learns the conditional score $\nabla_{z_{t}} lo g p (z_{t} ∣ c)$ (images given the prompt). Subtracting: $ϵ_{c} - ϵ_{\emptyset} \propto - [\nabla lo g p (z_{t} ∣ c) - \nabla lo g p (z_{t})] = - \nabla_{z_{t}} lo g p (c ∣ z_{t})$ by the Bayes identity. So their difference is (up to sign/scale) the implicit classifier gradient — with no separate classifier ever trained. The CLIP text embedding is what supplies $c$ .

Level 5 — Mastery

Exercise 5.1

You are given three unit embeddings in $R^{2}$ : image $v_{I} = (1, 0)$ , and two text candidates $v_{T, A} = (cos 3 0^{\circ}, sin 3 0^{\circ})$ and $v_{T, B} = (cos 8 0^{\circ}, sin 8 0^{\circ})$ . With $τ = 0.07$ , compute the zero-shot probability the image is class $A$ . Then state what happens to that probability as $τ \to \infty$ .

Recall Solution

Since all are unit vectors, cosine similarity is just the dot product. $s_{A} = cos 3 0^{\circ} \approx 0.86603$ , $s_{B} = cos 8 0^{\circ} \approx 0.17365$ . Logits: $s_{A} / τ = 0.86603/0.07 \approx 12.372$ , $s_{B} / τ \approx 2.481$ . $p (A) = \frac{e ^{12.372}}{e ^{12.372} + e ^{2.481}} = \frac{1}{1 + e ^{2.481 - 12.372}} = \frac{1}{1 + e ^{- 9.891}} \approx 0.99995.$ Almost certainly class $A$ — its arrow sits much closer to the image arrow ( $3 0^{\circ}$ vs. $8 0^{\circ}$ ). As $τ \to \infty$ : both logits $\to 0$ , so $p (A) \to \frac{e ^{0}}{e ^{0} + e ^{0}} = \frac{1}{2}$ . Infinite temperature erases all information — the classifier becomes a coin flip. (This is the mirror of Exercise 3.1.)

Exercise 5.2 (Design)

Combine everything: describe end-to-end how a text prompt becomes a guided denoising step, naming every module and where CLIP enters.

Recall Solution

Encode text. The prompt string goes through CLIP's text encoder $f_{T}$ (a transformer using attention) to produce $c = v_{T}$ .
Condition the U-Net. At each diffusion timestep $t$ , the noisy latent $z_{t}$ and the timestep are fed to the noise network. Cross-attention layers attend from image tokens to the CLIP text tokens $c$ — this is where the conditioning physically happens.
Two forward passes. Run the network once with $c$ ( $ϵ_{c}$ ) and once with the null embedding $\emptyset$ ( $ϵ_{\emptyset}$ ).
Guide. Form $\tilde{ϵ} = ϵ_{\emptyset} + w (ϵ_{c} - ϵ_{\emptyset})$ with e.g. $w = 7.5$ (Ex 4.1–4.3).
Denoise. Plug $\tilde{ϵ}$ into the reverse step $z_{t - 1} = \frac{1}{α _{t}} (z_{t} - \frac{1 - α _{t}}{1 - α ˉ _{t}} \tilde{ϵ}) + σ_{t} ϵ$ .
Repeat for all $t$ down to $0$ , then decode the latent to an image. CLIP appears exactly once — at step 1 — but its embedding is what steers every subsequent step. This ties together diffusion, guidance and multimodal learning.