Foundations — Text-to-image conditioning (CLIP)

Before you can read the parent note, you need to understand every piece of notation it throws at you. This page walks each symbol from absolute zero: what it means in plain words, what it looks like as a picture, and why CLIP needs it. Read top to bottom — each item leans on the one before.

1. A vector — an arrow with a list of numbers

Everything in CLIP is a vector. Forget the scary word: a vector is just an ordered list of numbers, and geometrically it is an arrow starting at the origin.

In CLIP every image and every sentence becomes such an arrow. Two arrows pointing the same way = "these mean the same thing". Two arrows pointing different ways = "these mean different things". That single geometric idea is the whole engine.

2. ${\mathbb{R}}^d$ — the space the arrows live in

The parent writes ${\mathbf{v}}_I\in {\mathbb{R}}^d$. Decode it piece by piece.

• $\mathbb{R}$ (blackboard R) means "all the real numbers" — every ordinary number like $3$, $-0.5$, $\pi$.
• ${\mathbb{R}}^d$ means "a list of $d$ real numbers", i.e. $d$-dimensional space.
• $\in$ means "is an element of" / "lives inside".

So ${\mathbf{v}}_I\in {\mathbb{R}}^{512}$ reads: "the image arrow is a list of 512 real numbers." We cannot draw 512 dimensions, so throughout we draw 2D or 3D and trust the same rules carry up.

3. The encoders $f_I(\cdot )$ and $f_T(\cdot )$ — the machines that make arrows

The subscripts are labels: I for image, T for text. Both machines are trained so their outputs land in the same ${\mathbb{R}}^d$, which is the only reason we are allowed to compare them later.

Why two separate machines? Images (grids of pixels) and text (sequences of words) are built completely differently, so each needs its own specialist reader. They rely on ideas from 4.4.7-Attention-mechanisms (how the text encoder reads a sentence) and image networks — but here we only care that each outputs an arrow.

4. The dot product $\mathbf{a}\cdot \mathbf{b}$ — measuring agreement

To ask "do these two arrows point the same way?" we need one number that measures agreement. That number is the dot product.

Why this tool and not another? We want one number that is large and positive when arrows point the same way, zero when they are perpendicular (unrelated), and negative when they point opposite ways. The dot product does exactly this because it equals $\Vert \mathbf{a}\Vert \Vert \mathbf{b}\Vert \cos \theta$ — it secretly hides the angle $\theta$ between the arrows.

5. The norm $\Vert \mathbf{v}\Vert$ — the length of an arrow

$\Vert \mathbf{v}\Vert$ (double bars) means the length of the arrow, found with the Pythagorean theorem:

$\Vert \mathbf{v}\Vert =\sqrt{v_1^2+v_2^2+\cdots +v_d^2}$

Picture the longest side of a right triangle whose other sides are the components. We need length because a long arrow and a short arrow can point the same direction — and for meaning we only care about direction, not length.

6. Cosine similarity — direction only

Now we combine the two: divide the dot product by both lengths.

Dividing by the lengths cancels magnitude, leaving pure direction. That is exactly why the parent says "we care about direction (semantic meaning), not magnitude." The superscripts $(i)$ and $(j)$ are just item numbers in a batch — image $i$, text $j$.

7. The subscript grid $s_{ij}$ — the similarity matrix

With $N$ pairs in a batch, $s_{ij}$ is the similarity of image $i$ to text $j$. Arranging every combination in a square grid gives the similarity matrix $S$.

The diagonal ($s_{11},s_{22},\dots$) holds the correct matches — image $i$ with its own caption $i$. Everything off the diagonal is a wrong pairing (a negative). CLIP's whole job: make the diagonal bright (high) and the rest dark (low). This idea comes straight from 4.3.8-Contrastive-learning.

8. Temperature $\tau$ — the sharpness dial (introduced first, because we need it next)

Before we can turn scores into probabilities, meet $\tau$ (Greek "tau"), a single positive number we divide every score by. It controls how decisive the final probabilities will be.

• Small $\tau$ (e.g. $0.07$): scores get divided into big numbers → the result will be peaky/confident.
• Large $\tau$: scores shrink → the result will be flat/uncertain.

Why a divide and not something else? Dividing scales all scores together, stretching or squashing the gaps between them without changing which is largest — a clean sharpness knob. CLIP even learns the best $\tau$ during training. Keep this in mind; it appears inside the formula in the very next section.

9. $\exp$, softmax, and probability — turning scores into confidence

Softmax uses $\exp$ (and the temperature $\tau$ from section 8) to turn a row of scores into probabilities that add to $1$:

$p(j\mid i)=\frac{\exp (s_{ij}/\tau )}{{\sum }_{k=1}^N\exp (s_{ik}/\tau )}$

• $p(j\mid i)$ reads "probability that text $j$ is the match, given image $i$" — the bar $\mid$ means "given".
• ${\sum }_{k=1}^N$ (capital sigma) means "add up over all $k$ from $1$ to $N$" — here, summing across every text in the row so the denominator is the total.
• $s_{ij}/\tau$ is each score divided by the temperature you just met.

Dividing each $\exp$ score by the sum guarantees the outputs are all positive and sum to exactly $1$ — a genuine probability distribution. We want $p(i\mid i)\approx 1$: image $i$ is confident its own caption is the match.

10. Cross-entropy loss $\mathcal{L}$ — the "how wrong are we" number

The minus sign flips $\log$ (which is negative for probabilities under $1$) into a positive penalty. $\mathcal{L}$ is the number training tries to shrink; shrinking it pushes correct pairs together and wrong pairs apart.

Why the loss must be symmetric (both directions)

So far $p(j\mid i)$ asks, for each image, "which text matches me?" — that is the image→text loss:

${\mathcal{L}}_{I\to T}=-\frac{1}{N}{\sum }_{i=1}^N\log p(i\mid i)$

But we can equally read down each column of the matrix and ask, for each text, "which image matches me?" — the text→image loss:

${\mathcal{L}}_{T\to I}=-\frac{1}{N}{\sum }_{j=1}^N\log p(j\mid j),p(i\mid j)=\frac{\exp (s_{ij}/\tau )}{{\sum }_{k=1}^N\exp (s_{kj}/\tau )}$

Note the denominator now sums over the column (all images $k$ for a fixed text $j$). CLIP averages the two so both encoders get equal pressure:

11. Conditioning symbols: $c$, $\varnothing$, $w$ (and the diffusion notation around them)

These appear when CLIP steers a diffusion model. First, the diffusion notation the parent borrows (built fully in 4.5.14-Diffusion-models):

Now the three steering symbols:

• $c$ = the text embedding ${\mathbf{v}}_T$ fed in as a condition — the "instruction".
• $\varnothing$ (empty set) = no condition — "generate anything".
• $w$ = the guidance scale, a knob for how hard to obey the text (typically $7$–$15$).

Putting it together, the difference

${ϵ}_{\theta }(z_t,t,c)-{ϵ}_{\theta }(z_t,t,\varnothing )$

is an arrow pointing from "any image" toward "an image matching the prompt": it is the guessed-noise-with-the-prompt minus the guessed-noise-with-no-prompt. Multiplying by $w$ stretches that arrow so the picture obeys the words more strongly. This is the exact bridge into 4.5.15-Classifier-free-guidance.

Prerequisite map

Each foundation feeds upward: arrows and lengths make cosine similarity, similarity fills the matrix, softmax and temperature turn it into probabilities, cross-entropy scores it in both directions, and the trained result unlocks guided generation. See also 4.2.12-Transfer-learning and 4.5.17-Multimodal-learning for where this leads.

Equipment checklist

Test yourself — cover the right side and answer before revealing.