Worked examples — Text-to-image conditioning (CLIP)

This page is the exercise floor for Text-to-image conditioning (CLIP). The parent note built the machinery — cosine similarity, temperature, contrastive loss, classifier-free guidance. Here we run the numbers through every awkward corner: matching pairs, mismatched pairs, temperature going to zero or infinity, guidance scale sweeping through $0$, $1$, and large values, and one exam-style trap.

Before we start, one promise: every symbol we touch was defined in the parent, but we re-anchor each to a picture so you never have to scroll back.

---

The scenario matrix

Every problem CLIP can throw at you lives in one of these cells. Throughout, $N$ denotes the number of items softmax is comparing — the batch size in the contrastive loss, or the number of candidate labels in zero-shot classification. The exam twist (Cell K) is the classic slip of feeding a raw, un-normalised dot product into the similarity slot: because it still carries the vectors' lengths, it escapes the $[-1,1]$ range and silently corrupts the temperature — the fix is to divide by both lengths first. The worked examples below are labelled by cell.

Cell	Scenario class	What's extreme about it	Example
A	Matching pair, sim near $+1$	positive pair, loss should be tiny	Ex 1
B	Mismatched pair, sim near $0$ or $-1$	negative pair, loss should be large	Ex 1
C	Full batch loss (both directions)	the real symmetric objective	Ex 2
D	Temperature $\tau \to 0$	softmax becomes a hard argmax	Ex 3
E	Temperature $\tau \to \infty$	softmax becomes uniform $1/N$ (with $N$ = number of items compared)	Ex 3
F	Guidance $w=0$	pure unconditional — prompt ignored	Ex 4
G	Guidance $w=1$	plain conditional, no amplification	Ex 4
H	Guidance $w>1$ (e.g. $7.5$)	extrapolation past the condition	Ex 4
I	Degenerate: identical embeddings	every similarity equal → tie	Ex 5
J	Real-world word problem	zero-shot classification decision	Ex 6
K	Exam twist: un-normalised vectors	raw dot product breaks the $[-1,1]$ range — "forgot to normalise"	Ex 7

---

Warm-up figure

Figure 1 above is the whole idea of cosine similarity in two dimensions. The angle $\theta$ between the blue image arrow and the pink text arrow is what we measure; $s=\cos \theta$. When they overlap, $\theta =0^{\circ }$ and $s=1$. When they're at a right angle, $s=0$. When they point opposite, $s=-1$.

---

Example 1 — Cells A & B: one positive, two negatives, by hand

Forecast: guess now — will ${\mathcal{L}}_1$ be close to $0$ (great match) or a big positive number? Because $s_{11}=0.90$ dominates the others, I predict a small loss.

1. Divide each similarity by $\tau$. Why this step? This is the temperature step of the softmax formula above. Small $\tau$ stretches the gaps, so the correct pair pulls far ahead. $\frac{s_{11}}{\tau }=9,\frac{s_{12}}{\tau }=1,\frac{s_{13}}{\tau }=-2$

2. Exponentiate. Why this step? Softmax works on $\exp (\cdot )$ so every score becomes positive and bigger scores blow up faster. $e^9\approx 8103.08,e^1\approx 2.71828,e^{-2}\approx 0.13534$

3. Normalise into a probability. Why this step? $p(1\mid 1)$ is the correct-text slice of the pie — numerator over the sum of all $N=3$ slices. $p(1\mid 1)=\frac{8103.08}{8103.08+2.71828+0.13534}=\frac{8103.08}{8105.94}\approx 0.99965$

4. Take negative log. Why this step? Cross-entropy loss is $-\log$ of the probability you put on the truth. ${\mathcal{L}}_1=-\log (0.99965)\approx 0.000353$

Verify: the loss is tiny and positive, matching the forecast — a confident correct match costs almost nothing. Cell A (the $0.90$ positive) dominated; cells B (the $0.10$ and $-0.20$ negatives) contributed almost nothing because their $\exp$ values are dwarfed. Good.

---

Example 2 — Cell C: the full symmetric batch loss

Forecast: the matrix is symmetric and diagonal-heavy, so both directions should give the same loss, and it should be small-ish but not zero.

1. Scale by $\tau$: every entry $\times 5$ → diagonal $=4$, off-diagonal $=0$. Why? Same temperature step as before; $1/\tau =5$.

2. Row softmax for image 1 (row $[4,0]$): Why? $p(1\mid 1)$ needs the row's own pie — here $N=2$ items. $p(1\mid 1)=\frac{e^4}{e^4+e^0}=\frac{54.598}{54.598+1}=0.98201$ By symmetry $p(2\mid 2)=0.98201$ too.

3. Image-to-text loss: Why this step? Using the definition above, ${\mathcal{L}}_{I\to T}$ is the average over the two images of each image's own cross-entropy $-\log p(i\mid i)$; the $-\frac{1}{2}$ is $-\frac{1}{N}$ with $N=2$, and both terms are equal here. ${\mathcal{L}}_{I\to T}=-\frac{1}{2}(\log 0.98201+\log 0.98201)=-\log 0.98201\approx 0.018150$

4. Text-to-image loss uses columns. Why? ${\mathcal{L}}_{T\to I}$ asks "given text $j$, which image?" — that reads down a column. The matrix is symmetric, so columns equal rows → ${\mathcal{L}}_{T\to I}=0.018150$.

5. Average the two directions: ${\mathcal{L}}_{\text{CLIP}}=\frac{1}{2}(0.018150+0.018150)=0.018150$

Verify: symmetric matrix ⇒ both directional losses equal ⇒ the average equals either one. Value $\approx 0.0182$, small because $s_{ii}=0.8\gg s_{ij}=0$. Consistent.

---

Example 3 — Cells D & E: temperature at both extremes

Figure 2 above plots the answer to this example: $p(1\mid 1)$ as $\tau$ sweeps from cold to hot.

Forecast: tiny $\tau$ should snap the probability toward the biggest entry ($s=0.9$), so $p\to 1$. Huge $\tau$ should flatten everything across all $N=3$ items, so $p\to 1/3$.

1. Case $\tau =0.05$ (approaching 0). Why this step? $1/\tau =20$ blows the gaps wide open. Scaled: $[18,10,8]$. Then $p(1\mid 1)=\frac{e^{18}}{e^{18}+e^{10}+e^8}=\frac{6.566\times 10^7}{6.566\times 10^7+2.203\times 10^4+2.981\times 10^3}\approx 0.999617$ Nearly 1 — the softmax became an almost-hard argmax. That is cell D.

2. Case $\tau =1000$ (approaching $\infty$). Why this step? $1/\tau =0.001$ crushes every gap to near zero. Scaled: $[0.0009,0.0005,0.0004]$. Then $p(1\mid 1)=\frac{e^{0.0009}}{e^{0.0009}+e^{0.0005}+e^{0.0004}}\approx 0.333400$ Almost exactly $1/3$ — the softmax became uniform over all $N=3$ items. That is cell E.

3. The limiting law. As $\tau \to 0^{+}$, softmax → one-hot on the max; as $\tau \to \infty$, softmax → uniform $1/N$ (here $1/3$). Figure 2 plots $p(1\mid 1)$ against $\tau$ so you can see the S-curve between these two shelves.

Verify: $0.999617$ hugs $1$ (argmax shelf), $0.333400$ hugs $1/3=0.3333$ (uniform shelf). Both limits confirmed. This is exactly why CLIP learns $\tau \approx 0.07$ — it wants sharp-but-not-brittle, sitting on the steep part of the curve.

---

Example 4 — Cells F, G, H: guidance scale sweep

Figure 3 above draws the guidance line and marks the three $w$ values worked below.

Forecast: $w=0$ ignores the prompt (should give $2.0$); $w=1$ gives the plain conditional ($5.0$); $w=7.5$ overshoots past $5.0$.

1. Find the guidance direction. Why this step? The bracket $[{ϵ}_c-{ϵ}_{\varnothing }]$ is the arrow "toward the prompt". See Figure 3 — it is the pale-yellow arrow from the unconditional dot to the conditional dot. ${ϵ}_c-{ϵ}_{\varnothing }=5.0-2.0=3.0$

2. Cell F, $w=0$: $\widetilde{ϵ}=2.0+0\cdot 3.0=2.0$ The prompt is entirely ignored — you get generic images. This is why $w=0$ is useless for text-to-image.

3. Cell G, $w=1$: $\widetilde{ϵ}=2.0+1\cdot 3.0=5.0$ Exactly the conditional prediction — faithful but low-contrast, often washed-out.

4. Cell H, $w=7.5$: $\widetilde{ϵ}=2.0+7.5\cdot 3.0=2.0+22.5=24.5$ We extrapolate far beyond ${ϵ}_c$. This is extrapolation, not interpolation — that is what makes samples look punchy and prompt-obedient, but push $w$ too high and they oversaturate.

Verify: the three answers $2.0,5.0,24.5$ lie on the straight line $\widetilde{ϵ}=2+3w$; Figure 3 shows them as three chalk dots on that line. Sanity: at $w=1$ we exactly recover ${ϵ}_c$, which the formula must do (plug $w=1$: ${ϵ}_{\varnothing }+({ϵ}_c-{ϵ}_{\varnothing })={ϵ}_c$). Correct. For the derivation of why the bracket approximates the classifier gradient, see 4.5.15-Classifier-free-guidance.

---

Example 5 — Cell I: degenerate identical embeddings (the tie)

Forecast: every score equal ⇒ softmax can't tell them apart ⇒ perfect three-way tie ⇒ $p=1/3$, and the loss should be $\log 3$.

1. Scale and exponentiate. Why? $s/\tau =[8.571,8.571,8.571]$; all three $\exp$ values are identical, call it $E$.

2. Softmax collapses to uniform. Why? $p=E/(E+E+E)=1/3$ regardless of $E$. The temperature cancels — a tie stays a tie at any temperature. $p(1\mid 1)=\frac{1}{3}\approx 0.33333$

3. Loss: ${\mathcal{L}}_1=-\log \frac{1}{3}=\log 3\approx 1.098612$

Verify: $\log 3=1.0986$ is exactly the cross-entropy of a uniform guess over 3 classes — the maximum-confusion loss for $N=3$. This is the sanity floor every fresh CLIP run starts near: at initialisation all embeddings are near-random, similarities are roughly equal, and the loss sits near $\log N$. If your training loss starts far above $\log N$, something is wrong. Degenerate case handled.

---

Example 6 — Cell J: real-world zero-shot classification decision

Forecast: dog has the highest similarity, so dog wins; with $\tau =0.07$ the gaps get amplified, so confidence should be high — I'd guess $>0.6$.

1. Scale by $1/\tau =14.2857$. Why? Same conditioning step. Scaled logits: $[0.31,0.25,0.10]\times 14.2857=[4.4286,3.5714,1.4286]$

2. Exponentiate: $[83.80,35.57,4.173]$. Why? Softmax numerators.

3. Normalise: $p(\text{dog})=\frac{83.80}{83.80+35.57+4.173}=\frac{83.80}{123.54}\approx 0.678$ $p(\text{cat})\approx \frac{35.57}{123.54}\approx 0.288,p(\text{bird})\approx \frac{4.173}{123.54}\approx 0.034$

4. Decision: pick $\arg \max =$ dog with confidence $\approx 0.678$.

Verify: probabilities sum to $0.678+0.288+0.034=1.000$ ✓ (must, since softmax). Dog beats cat by a hair in raw similarity ($0.31$ vs $0.25$) but temperature stretches it to a $68\%$ vs $29\%$ gap — that is exactly what low $\tau$ buys you. This is the mechanism behind zero-shot transfer: no training on this task, just similarity + softmax.

---

Example 7 — Cell K: the "forgot to normalise" exam trap

Forecast: the correct cosine must land in $[-1,1]$; since the two vectors are close-ish in direction, expect something like $0.9$–$1.0$.

1. Dot product (the student's number): Why? $\mathbf{a}\cdot \mathbf{b}=3\cdot 4+4\cdot 3=12+12=24$. Correct arithmetic, wrong measure — it still carries the lengths.

2. Compute the two lengths. Why this step? Cosine strips length; we must divide it out. $\Vert {\mathbf{v}}_I\Vert =\sqrt{3^2+4^2}=\sqrt{25}=5,\Vert {\mathbf{v}}_T\Vert =\sqrt{4^2+3^2}=5$

3. Divide: $s=\frac{24}{5\cdot 5}=\frac{24}{25}=0.96$

4. Diagnose the trap. The dot product $24$ was outside $[-1,1]$ purely because the arrows are long ($5$ units each). Cosine's denominator $\Vert {\mathbf{v}}_I\Vert \Vert {\mathbf{v}}_T\Vert =25$ exactly cancels that length, returning the honest direction-agreement $0.96$.

Verify: $0.96\in [-1,1]$ ✓. Cross-check via the angle: ${\cos }^{-1}(0.96)\approx 16.26^{\circ }$, a small angle — matching the fact that $(3,4)$ and $(4,3)$ are near-mirror-images that point in nearly the same direction. Never feed raw dot products into the CLIP softmax; always normalise first, or your temperature is silently wrong.

---

Recall Self-test

At $\tau \to \infty$, what does $p(1\mid 1)$ approach for a batch of $N$? ::: $1/N$ (uniform) — see Ex 3, cell E. A perfectly-tied similarity row gives what loss for $N=3$? ::: $\log 3\approx 1.0986$ — Ex 5, cell I. With ${ϵ}_{\varnothing }=2$, ${ϵ}_c=5$, $w=7.5$, what is $\widetilde{ϵ}$? ::: $24.5$ — Ex 4, cell H. For raw vectors $(3,4)$ and $(4,3)$, the cosine similarity is? ::: $0.96$ — Ex 7, cell K.

Where to go next: the attention-based text encoder that produces these embeddings lives in 4.4.7-Attention-mechanisms; the broader picture of joining modalities is 4.5.17-Multimodal-learning; the diffusion sampler that consumes $\widetilde{ϵ}$ is in 4.5.14-Diffusion-models.