Text-to-image conditioning (CLIP): Visual walkthrough

Everything below assumes only: you can measure the angle between two arrows, and you know that a bigger number beats a smaller number. That's it.

Step 1 — Turn a picture and a sentence into arrows

WHAT. A neural network reads a photo and spits out a list of $d$ numbers. A different network reads a sentence and spits out its own list of $d$ numbers. A list of $d$ numbers is an arrow in $d$ -dimensional space (a vector). We call the image arrow $v_{I}$ and the text arrow $v_{T}$ .

WHY. We cannot compare a JPEG to English directly — they live in different worlds. But if both become arrows in the same space, we can ask a single simple question: do these two arrows point the same way? That shared space is the whole trick.

PICTURE. Below, a dog photo becomes the cyan arrow, the words "a dog" become the amber arrow. Right now they point in random directions — training hasn't happened yet.

Now let us name the two machines that do this arrow-making. "The image network" is a mouthful, so we give it a short name: $f_{I}$ . It is just a label for the whole image-reading network — feed it an image $x$ , and out comes the arrow. We write that as $f_{I} (x) = v_{I}$ , read aloud as " $f_{I}$ of $x$ equals $v_{I}$ ", i.e. "run image $x$ through the image network and you get the image arrow." The parentheses just mean "here is the input I'm feeding in." Likewise $f_{T}$ is the label for the whole sentence-reading network, and $f_{T} (t) = v_{T}$ means "run sentence $t$ through the text network and you get the text arrow." The subscript $I$ always means image, $T$ always means text — that's all $f_{I}$ and $f_{T}$ are: shorthand names for the two networks from the picture above.

Here $v_{I}$ = "where the image landed", and $v_{T}$ = "where the words landed".

Step 2 — Measure "same direction" with the cosine

WHAT. To score how aligned two arrows are, we use the cosine of the angle between them. If the angle is $0°$ they point identically → cosine $= 1$ . If they are perpendicular ( $90°$ ) → cosine $= 0$ . If they point oppositely ( $180°$ ) → cosine $= - 1$ .

WHY this tool and not another? We could measure the straight-line distance between the arrow tips, but that gets fooled by length. A long arrow and a short arrow can point the same way yet be far apart. We only care about direction (the meaning), never how long the network happened to make the arrow. The cosine throws length away and keeps only direction — exactly what we want.

The cosine is computed from the dot product (multiply matching entries, add them up) divided by the two lengths:

s_{ij} = s_{ij} = c o s (angle) \in [- 1, 1] \frac{v _{I}^{(i)} \cdot v _{T}^{(j)} dot product: alignment \times lengths}{divide the lengths back out ∥ v _{I}^{(i)} ∥ ∥ v _{T}^{(j)} ∥}

$s_{ij}$ — the similarity score between image $i$ and text $j$ . Big = they agree.
$v_{I}^{(i)}$ — arrow of the $i$ -th image in the batch; the superscript $(i)$ is just "which one".
$∥ \cdot ∥$ — the length of an arrow (its magnitude).

PICTURE. Same arrow pair, now with the angle $θ$ marked and the cosine dial from $- 1$ to $+ 1$ .

Step 3 — Lay every pair on a grid: the similarity matrix

WHAT. Take a batch of $N$ pairs. Score every image against every text. That fills an $N \times N$ grid called the similarity matrix $S$ . Formally,

S \in R^{N \times N}, S_{ij} = s_{ij} = cos (v_{I}^{(i)}, v_{T}^{(j)})

$S$ — the whole grid of scores; $N$ rows (images), $N$ columns (texts).
$S_{ij}$ — the entry in row $i$ , column $j$ : the similarity of image $i$ with text $j$ .

The diagonal ( $i = j$ ) holds the true matches; everything off-diagonal ( $i \neq = j$ ) holds mismatches.

WHY. To teach "these two belong together" you must also show counter-examples of things that don't. The off-diagonal cells are the free negatives — image 1 vs text 2, image 1 vs text 3, and so on. See 4.3.8-Contrastive-learning for why negatives are the engine of this whole idea.

PICTURE. A $3 \times 3$ grid. Amber diagonal = the pairs we want high. Faded cyan off-diagonal = the pairs we want low.

Step 4 — Sharpen the scores with temperature $τ$

WHAT. Divide every score by a small positive number $τ$ (tau): $logits_{ij} = s_{ij} / τ$ .

WHY. Cosine scores are squashed into $[- 1, 1]$ — a winning $0.9$ and a losing $0.3$ are barely different numbers. In the next step we exponentiate them, and small gaps make weak decisions. Dividing by a small $τ \approx 0.07$ stretches the gaps: $0.9/0.07 \approx 12.86$ vs $0.3/0.07 \approx 4.29$ — the winner-to-loser distance grows from $0.6$ to about $8.57$ , so the winner now towers over the loser. $τ$ is a dial for confidence: small $τ$ = sharp/decisive, large $τ$ = soft/hesitant. CLIP even learns the best $τ$ .

logits_{ij} = \frac{s _{ij}}{τ}, τ > 0

$logits_{ij}$ — the stretched score, feeding the exponential next.
$τ$ — temperature; smaller = sharper. Named after physics: cold systems are decisive.

PICTURE. One row of scores before and after dividing by $τ$ — the tall winner emerges.

Step 5 — Turn a row of scores into probabilities (softmax)

WHAT. For image $i$ , we want a probability for each candidate text $j$ : "how likely is text $j$ the true caption?" The recipe: exponentiate every stretched score, then divide by their total. This recipe has a standard name — it is called the softmax function, and any time you see "softmax" from now on it means exactly this "exponentiate then divide by the sum" operation.

WHY exponentiate? Two reasons in one move. (1) $exp$ is always positive, so we get valid probabilities (no negatives). (2) $exp$ grows fast, so a slightly larger logit becomes a much larger share — the winner-takes-most behaviour we sharpened for in Step 4. Dividing by the row total forces the probabilities to sum to $1$ .

p (j ∣ i) = \frac{exp ( s _{ij} / τ ) this cell’s weight}{total weight of the whole row k = 1 \sum N exp ( s _{ik} / τ )}

$p (j ∣ i)$ — probability the model assigns text $j$ to image $i$ ; the bar means "given image $i$ ".
$exp (\cdot)$ — the exponential, our positive-and-amplifying function.
$\sum_{k = 1}^{N}$ — add over all texts in the row (index $k$ sweeps $1 \dots N$ ); this is the normaliser.
The whole right-hand side, taken together, is the softmax of row $i$ .

PICTURE. The row of logits becomes a bar chart of probabilities that sum to a full bar of height $1$ .

Step 6 — Score the guess: cross-entropy loss

WHAT. We know the right answer for image $i$ : it is text $i$ (the diagonal). We want $p (i ∣ i)$ to be $1$ . The loss for image $i$ is $- lo g p (i ∣ i)$ . Average over the batch to get one number.

WHY the $- lo g$ ? A perfect guess $p (i ∣ i) = 1$ gives $- lo g 1 = 0$ — zero penalty, perfect. A terrible guess $p (i ∣ i) \to 0$ gives $- lo g (tiny) \to + \infty$ — huge penalty. So $- lo g$ is a pain-meter: no pain when right, unbounded pain when confidently wrong. This is exactly cross-entropy against a one-hot target (all the truth sits on cell $i$ ).

L_{I \to T} = - \frac{1}{N} i = 1 \sum N lo g \frac{exp ( s _{ii} / τ )}{j = 1 \sum N exp ( s _{ij} / τ )}

$L_{I \to T}$ — the image→text loss: each image must find its own caption.
$s_{ii}$ — the diagonal score, the right answer we want on top.
$- \frac{1}{N} \sum_{i}$ — average the pain over all $N$ images.

Both directions. By symmetry we also make each text find its own image, giving $L_{T \to I}$ (same formula but summing down columns). The final CLIP loss averages the two:

L_{CLIP} = \frac{1}{2} (L_{I \to T} + L_{T \to I})

Training both directions stops the model from lazily solving only one side. Under the hood this is transfer learning fuel: the encoders end up general enough to reuse anywhere.

PICTURE. The pain-meter curve $- lo g p$ : flat at zero when right, shooting to infinity when wrong.

Step 7 — Edge cases: what happens at the extremes

WHAT & WHY. A derivation is only trustworthy if it survives its boundaries. Walk each one:

Perfect batch — diagonal cosines $= 1$ , off-diagonal $= - 1$ . After $/ τ$ the diagonal explodes upward, $p (i ∣ i) \to 1$ , loss $\to 0$ . Nothing left to learn — correct.
Random batch (start of training) — every score $\approx 0$ , so every probability $\approx 1/ N$ . Then loss $\approx - lo g (1/ N) = lo g N$ . For $N = 3$ that is $lo g 3 \approx 1.0986$ . This is the baseline you should see on step 0.
Duplicate captions — if texts $i$ and $j$ are identical, the model cannot separate them and the loss can never reach $0$ . Not a bug; it's the data being ambiguous.
Batch size $N = 1$ — one pair, no negatives, denominator is a single term, so $p (1 ∣ 1) = 1$ always and loss $= 0$ regardless. The model learns nothing. This is why CLIP needs large batches — negatives are the teacher.

PICTURE. Three mini-grids: perfect (bright diagonal), random (uniform grey), and $N = 1$ (a single lit cell that teaches nothing).

Step 8 — From trained CLIP to steering an image generator

First, three brand-new symbols. In this step we leave CLIP and hand its text arrow to a diffusion generator, which introduces things we have not met yet.

What is $θ$ ? The diffusion network, like any neural network, is a giant bag of adjustable numbers (its weights) that training slowly tunes. We bundle all of those tunable numbers into one symbol, $θ$ (theta), and read it as "the model's parameters." Writing $ϵ_{θ}$ (epsilon-sub-theta) just means "the noise-prediction produced by the network whose weights are $θ$ " — the subscript reminds us this output depends on the trained weights, not on some fixed formula. So $ϵ_{θ} (z_{t}, t, c)$ is "run the trained network on the noisy image $z_{t}$ at step $t$ , told about condition $c$ , and read off its guess for the noise."

WHAT. Once CLIP's arrows are aligned, feed the text arrow $c = v_{T}$ into the diffusion model as the condition. At each denoising step the model predicts the noise in $z_{t}$ two ways — with the prompt and without — and combines them:

\tilde{ϵ} = plain guess, no prompt ϵ_{θ} (z_{t}, t, \emptyset) + w \cdot arrow that points toward the prompt [ϵ_{θ} (z_{t}, t, c) - ϵ_{θ} (z_{t}, t, \emptyset)]

$z_{t}$ — the noisy latent (half-cleaned image) at step $t$ .
$t$ — the diffusion time step (which round of denoising we are on).
$θ$ — all the trained weights of the diffusion network.
$ϵ_{θ}$ — the diffusion network's predicted noise inside $z_{t}$ .
$c$ — the CLIP text arrow; $\emptyset$ — empty prompt.
$w$ — guidance scale; how hard we lean toward the prompt.
The bracket is a direction: "conditional minus unconditional" = the way the words tug the image.

WHY. The difference isolates the prompt's pull; scaling it by $w$ amplifies that pull. This is exactly classifier-free guidance riding on top of the diffusion denoiser, using attention to inject $c$ .

PICTURE. A vector diagram: unconditional noise as one arrow, the prompt-direction added and stretched by $w$ to reach the guided noise $\tilde{ϵ}$ .

The one-picture summary

The figure below is the whole page as a labelled conveyor belt: each stage is a box, and above every arrow between boxes is written the exact quantity that flows out of it — so you can watch the data change form step by step rather than reading a bare list of names. Narrate it out loud from the left: two inputs (image $x$ , text $t$ ) enter; the two networks $f_{I}, f_{T}$ emit arrows $v_{I}, v_{T}$ (Step 1); scoring every pair with the cosine yields the grid $S$ (Steps 2–3); dividing by $τ$ produces logits $s / τ$ (Step 4); softmax turns each row into probabilities $p (j ∣ i)$ (Step 5); $- lo g$ of the diagonal gives the loss $L_{CLIP}$ (Step 6); and the now-aligned text arrow becomes the condition $c$ that steers the generator (Step 8). Every symbol you met is exactly one label on this belt — if you can name each arrow's cargo, you own the derivation.

Recall Feynman retelling — say it back in plain words

We hand a computer a photo and a sentence. Two networks, $f_{I}$ and $f_{T}$ , squeeze each into an arrow living in the same room. To ask "do these mean the same thing?" we measure the angle between the arrows and take its cosine — $1$ for same direction, $- 1$ for opposite — and we ignore how long the arrows are because length isn't meaning. In a batch of $N$ pairs we score every photo against every sentence, filling a grid whose diagonal is the true couples. We divide every score by a tiny number $τ$ to stretch the gaps, then run each row through softmax so the scores become probabilities that add to one. We punish the model with $- lo g$ of the probability it put on the correct partner: no pain when it's sure and right, infinite pain when it's sure and wrong. We do this both ways — photos finding sentences and sentences finding photos — and average. Slowly the grid turns bright-diagonal, dark-elsewhere: matching arrows now point together. Finally we hand a sentence-arrow to an image generator whose weights are called $θ$ , which computes "with prompt minus without prompt" to get a direction pointing toward the words, and cranks that direction up by a knob $w$ to paint exactly what we asked for.

Recall Quick checks

What does dividing the dot product by both lengths accomplish? ::: It removes arrow length, leaving only direction — i.e. it turns the dot product into the cosine, so only meaning is scored, not magnitude. Why exponentiate the scores before normalising? ::: $exp$ is always positive (valid probabilities) and grows fast (amplifies the winner), giving a decisive winner-take-most distribution. At the very start of training, what loss value should you see for batch size $N$ ? ::: About $lo g N$ , because every probability is roughly $1/ N$ and $- lo g (1/ N) = lo g N$ . In guidance, what does the bracket $ϵ_{θ} (z_{t}, t, c) - ϵ_{θ} (z_{t}, t, \emptyset)$ represent? ::: A direction vector pointing from "no prompt" toward "the prompt" — the pull the text exerts, which $w$ then amplifies. What does the subscript $θ$ in $ϵ_{θ}$ stand for? ::: All the trained weights (parameters) of the diffusion network; it reminds us the noise prediction depends on those learned numbers. Why can't CLIP learn with batch size 1? ::: With only one pair there are no negatives, so the softmax denominator is a single term, forcing $p (1 ∣ 1) = 1$ and loss $= 0$ every time — zero gradient, so the encoders never get pushed and nothing is learned.