Foundations — Multimodal models (vision-language)

This page is the toolbox. The parent note freely uses vectors, dimensions, dot products, cosine, softmax, logarithms and temperature. If any of those made you pause, start here. We define each symbol in plain words, draw its picture, and say why the topic needs it — in an order where each brick rests on the one before.

1. What is a vector? (the arrow)

Before anything, we need to know what an arrow in space is, because the whole topic lives on arrows.

Look at the figure. The pair $(3,1)$ is not "two separate numbers" — it is one arrow: go 3 right, 1 up, draw from origin to there. The red arrow is that vector.

Why the topic needs this. The parent note writes $v_{\text{img}}\in {\mathbb{R}}^d$ and $v_{\text{text}}\in {\mathbb{R}}^d$. Read that as: "the image becomes one arrow, the sentence becomes another arrow." An image of a cat and the word "cat" are just two arrows we hope point the same direction.

$\in$ means "is a member of". So $v_{\text{img}}\in {\mathbb{R}}^d$ reads: "$v_{\text{img}}$ is one of the arrows in the $d$-number space."

2. Length of a vector (how long the arrow is)

Now that we have an arrow, we ask: how long is it? We need this because cosine similarity later divides by lengths.

The general rule: square every component, add them, take the square root.

$\Vert v\Vert =\sqrt{v_1^2+v_2^2+\cdots +v_d^2}$

Why the topic needs it. The parent's cosine formula has $\Vert v_{\text{img}}\Vert \Vert v_{\text{text}}\Vert$ in the denominator. That division is what makes the model care about direction only, not how long the arrows happen to be — a short "cat" arrow and a long "cat" arrow should still count as the same concept.

3. The dot product (do two arrows agree?)

This is the heart of the whole comparison. We need a single number that says "do these two arrows point the same way?"

Here is the magic picture behind it:

The dot product secretly equals $\Vert a\Vert \Vert b\Vert \cos \theta$, where $\theta$ (the Greek letter "theta") is the angle between the two arrows. So the sign of the dot product tells you the geometry:

• Arrows point the same way (small angle) → dot product large and positive.
• Arrows are perpendicular (90°) → dot product exactly zero.
• Arrows point opposite (angle over 90°) → dot product negative.

Why the topic needs it. "Matching image and text should be close" literally means "their arrows should have a large positive dot product / small angle." The dot product is the raw agreement score; cosine (next) just cleans it up.

4. Cosine similarity (agreement, size removed)

The raw dot product mixes two things: the angle AND the lengths. We want angle alone. So we divide the lengths out.

Because we divided out both lengths, all that survives is $\cos \theta$ — a number that depends only on the angle $\theta$.

The figure shows the cosine of the angle as it sweeps from $0°$ to $180°$:

• $\theta =0°$ → $\cos \theta =+1$ → identical direction (same concept).
• $\theta =90°$ → $\cos \theta =0$ → unrelated.
• $\theta =180°$ → $\cos \theta =-1$ → opposite.

That is exactly the "$[-1,1]$, +1 = same, 0 = orthogonal, −1 = opposite" table in the parent note. Now you know why those numbers appear: they are cosines of angles between arrows.

5. Exponential and logarithm (turning scores into confidence)

The training loss uses $\exp$ and $\log$. These are two tools that undo each other, and each earns its place.

Why the topic needs it. Similarities can be negative or zero, but a probability must be positive. Feeding a similarity through $\exp$ guarantees a positive number — and because $\exp$ grows so fast, a slightly higher similarity becomes a much higher value. That is why the parent's example turned $0.9$ vs $0.1$ into near-certainty $0.9999$: $\exp$ amplifies the winner.

Why the topic needs it. The loss is $-\log (\text{probability of the correct pair})$. If the model is confident (probability near 1), $\log$ of it is near 0, so the loss is near 0 — "you did well, nothing to fix." If the probability is tiny, $\log$ is a big negative number, and the minus sign makes the loss big — "you were wrong, fix a lot." The $-\log$ turns confidence into penalty. This is the cross-entropy loss in disguise.

6. Softmax (turning scores into a probability list)

The parent asks the model to pick the correct caption out of a batch. Softmax is the tool that converts a row of similarity scores into a list of probabilities that sum to 1.

Read the pieces you now know:

• $\exp (s_i)$ — make each score positive (Section 5).
• ${\sum }_{j=1}^B$ — the big Greek "S" ($\Sigma$, sigma) means "add up over all $j$ from 1 to $B$", so the denominator is the total.
• Dividing each top by the total → every $P_i$ lands in $[0,1]$ and they all add to 1: a real probability distribution.

Why the topic needs it. Contrastive training treats "which of the $B$ captions matches this image?" as a multiple-choice question. Softmax turns the $B$ similarity scores into the probability the model assigns to each choice. Then $-\log$ of the probability on the correct choice is the loss.

Picture two knobs on the same dial: turn $\tau$ down for a decisive model, up for a hesitant one.

7. Batch, sum, and matrix notation (the bookkeeping)

• Batch size $B$ — how many (image, text) pairs we process together in one step. In the parent example $B=4$.
• Similarity matrix — a $B\times B$ grid where entry (row $i$, column $j$) is $\text{sim}(v_i,v_{t_j})$: image $i$'s arrow against caption $j$'s arrow. The diagonal (row $i$, column $i$) holds the correct matches; everything off the diagonal is a wrong pairing we want to push apart.
• Subscripts like $v_i$, $t_j$, ${\text{PE}}_i$ — just labels ("the $i$-th one"), not multiplication.
• $W_v,W_t$ — projection matrices, learned tables of numbers that reshape an encoder's output arrow into the shared $d$-dimensional space. Think of them as adapters plugging two different cables into one common socket. (See transfer learning for why we reuse pre-trained encoders and only learn these small adapters.)

Prerequisite map

The two encoders (image and text) sit on transformer architecture and Vision Transformers, learn via self-supervised learning on paired data, and rest on the idea of word embeddings — words as arrows — extended so that pictures become arrows too. The comparison machinery leans on attention inside the encoders and feeds forward to text-to-image generation.

Worked micro-check

Let $v_{\text{img}}=(3,1)$ and $v_{\text{text}}=(2,2)$.

1. Dot product: $3\cdot 2+1\cdot 2=8$.
2. Lengths: $\Vert v_{\text{img}}\Vert =\sqrt{10}$, $\Vert v_{\text{text}}\Vert =\sqrt{8}$.
3. Cosine: $\frac{8}{\sqrt{10}\sqrt{8}}=\frac{8}{\sqrt{80}}\approx 0.894$.

An angle whose cosine is $0.894$ is about $26.6°$ — the arrows nearly agree, so this image and text would be scored a strong match.

Equipment checklist

Cover the right side; can you answer before revealing?