Worked examples — Multimodal models (vision-language)

This page is the worked-example lab for the parent topic. The parent built the machinery (encoders, the attention-based transformer encoders, cosine similarity, the contrastive loss). Here we run that machinery by hand across every case it can throw at you — every sign of similarity, every degenerate input, every limit of the temperature, plus a word problem and an exam twist.

Everything is built from zero. If a symbol shows up, it was defined first.

---

The scenario matrix

Every question this topic asks reduces to comparing arrows with cosine similarity and squeezing those comparisons through a softmax with a temperature. Here is every case class that can appear:

Cell	Case class	What's special	Hit by
A	Positive alignment	arrows point similar way, sim $>0$	Ex 1
B	Orthogonal	arrows at right angles, sim $=0$	Ex 2
C	Opposite	arrows point away, sim $<0$	Ex 2
D	Zero / degenerate vector	one arrow has length $0$ — sim undefined	Ex 3
E	Magnitude invariance	scaling an arrow must not change sim	Ex 4
F	Softmax + temperature (normal)	full loss on a batch, $\tau \approx 0.07$	Ex 5
G	Limit $\tau \to 0$	sharpening to a hard argmax	Ex 6
H	Limit $\tau \to \infty$	flattening to pure uncertainty	Ex 6
I	Real-world word problem	zero-shot retrieval decision	Ex 7
J	Exam twist	noisy-pair robustness / symmetric loss trap	Ex 8

We now cover every cell.

---

Reusable tool — cosine similarity, defined from the picture

---

Worked examples

Example 1 — Cell A: positive alignment

Forecast: guess before computing — both arrows lean up-and-right, close together. Do you expect sim near $+1$, near $0$, or near $-1$?

1. Dot product: $v_{\text{img}}\cdot v_{\text{dog}}=(3)(4)+(4)(3)=24.$ Why this step? The dot product is the numerator — it measures raw agreement coordinate by coordinate.
2. Lengths: $\Vert v_{\text{img}}\Vert =\sqrt{3^2+4^2}=\sqrt{25}=5$, and $\Vert v_{\text{dog}}\Vert =\sqrt{4^2+3^2}=5.$ Why this step? We must strip out size so only direction survives.
3. Similarity: $\text{sim}=\frac{24}{5\cdot 5}=\frac{24}{25}=\mathrm{0.96.}$ Why this step? Dividing by the lengths converts "agreement" into a pure $\cos \theta$.

Verify: $0.96$ is in $[-1,1]$ ✓ and close to $+1$, matching the forecast that the arrows nearly overlap. The angle is $\arccos (0.96)\approx 16.3°$ — a small gap, as drawn.

---

Example 2 — Cells B & C: orthogonal and opposite

Forecast: which one gives exactly $0$, and which gives exactly $-1$?

Part (B) orthogonal:

1. Dot: $(3)(4)+(4)(-3)=12-12=0.$ Why? A dot product of zero is the algebraic signature of a right angle.
2. sim $=0/(5\cdot 5)=0.$ Cell B done — completely unrelated concepts.

Part (C) opposite:

1. Dot: $(3)(-3)+(4)(-4)=-9-16=-25.$ Why? Every coordinate disagrees in sign, so the sum is fully negative.
2. Lengths: $\Vert v_{\text{anti}}\Vert =\sqrt{9+16}=5.$
3. sim $=-25/(5\cdot 5)=-1.$ Cell C done — the arrows point $180°$ apart.

Verify: angles are $\arccos (0)=90°$ ✓ and $\arccos (-1)=180°$ ✓. Both fall in $[-1,1]$. Note: negative similarity is allowed and meaningful — it says "actively push these apart."

---

Example 3 — Cell D: the degenerate zero vector

Forecast: will the formula give $0$, $1$, or something forbidden?

1. Dot: $(0)(4)+(0)(3)=0.$
2. Length: $\Vert v_{\text{img}}\Vert =\sqrt{0+0}=0.$
3. sim $=\frac{0}{0\cdot 5}=\frac{0}{0}$ — undefined (division by zero). Why this matters: a zero arrow has no direction, so "which way does it point?" has no answer.

Verify: you cannot report a number. In practice code either (a) adds a tiny $\epsilon$ so $\Vert v\Vert =\sqrt{\epsilon }$ and sim $\to 0$, or (b) rejects the batch. The lesson: always guard against zero-length embeddings — they crash cosine similarity. This is the single most common silent bug when training these models with cross-entropy-style contrastive loss.

---

Example 4 — Cell E: magnitude invariance

Forecast: guess whether the two similarities are equal or different.

1. Original: from Example 1, $\text{sim}(v_{\text{img}},v_{\text{dog}})=24/25=\mathrm{0.96.}$
2. Scaled dot: $(3)(40)+(4)(30)=120+120=240.$
3. Scaled length: $\Vert v_{\text{dog}}^{\prime }\Vert =\sqrt{40^2+30^2}=\sqrt{2500}=50.$
4. Scaled sim: $240/(5\cdot 50)=240/250=\mathrm{0.96.}$

Verify: identical — $0.96=0.96$ ✓. Why this step mattered: it proves cosine ignores magnitude. This is exactly why we chose cosine (not the raw dot product): a concept's identity is its direction, and transfer to unseen data relies on that stability.

---

Reusable tool — softmax with temperature, defined from zero

---

Example 5 — Cell F: full softmax + loss, normal temperature

Forecast: with a gap of $0.7$ and small $\tau$, do you expect $P(\text{dog})$ near $0.5$ or near $1$?

1. Scale by $\tau$: $0.9/0.1=9$, $0.2/0.1=2$, $0.0/0.1=0.$ Why this step? Dividing by $\tau =0.1$ multiplies the gaps tenfold — this is where "confidence" is dialed in.
2. Exponentiate: $e^9\approx 8103.08$, $e^2\approx 7.389$, $e^0=1.$ Why? $\exp$ makes all terms positive and explodes the leader.
3. Sum: $8103.08+7.389+1\approx \mathrm{8111.47.}$
4. $P(\text{dog})=8103.08/8111.47\approx \mathrm{0.99897.}$
5. Loss: $-\log (0.99897)\approx \mathrm{0.00103.}$

Verify: $P$ is in $[0,1]$ ✓, the three probabilities sum to $1$ ✓, and the tiny loss confirms a confident, correct model. Matches the forecast: near $1$.

---

Example 6 — Cells G & H: the two temperature limits

Forecast: which limit turns softmax into a hard "winner takes all," and which into "everyone equal"?

Part (G) $\tau =0.01$ (sharpening):

1. Scale: $90,20,0$. The gaps are now huge.
2. $P(\text{dog})=\frac{e^{90}}{e^{90}+e^{20}+e^0}\approx 1.0000$ (the others are astronomically smaller). Why? As $\tau \to 0$ the largest score dominates completely — softmax becomes argmax (a hard pick). Cell G.

Part (H) $\tau =1000$ (flattening):

1. Scale: $0.0009,0.0002,0.0$. Gaps are crushed to near-zero.
2. $e^{0.0009}\approx 1.0009$, $e^{0.0002}\approx 1.0002$, $e^0=1$. Sum $\approx 3.0011$.
3. $P(\text{dog})=1.0009/3.0011\approx \mathrm{0.33352.}$ Why? As $\tau \to \infty$ every score looks equal — softmax $\to$ uniform $1/B=1/3$. Cell H.

Verify: (G) gives $\approx 1$, (H) gives $\approx 1/3\approx 0.3333$ ✓ — exactly the two extremes. This is why $\tau \approx 0.07$ is chosen and even learned: it sits between "over-confident" and "clueless."

---

Example 7 — Cell I: real-world word problem (zero-shot retrieval)

Forecast: guess the winner from the arrows (all in the same up-right region wins).

1. Lengths: $\Vert v_{\text{img}}\Vert =\sqrt{36+64}=10$; $\Vert [5,10]\Vert =\sqrt{125}\approx 11.180$; $\Vert [9,1]\Vert =\sqrt{82}\approx 9.055$; $\Vert [-2,-1]\Vert =\sqrt{5}\approx \mathrm{2.236.}$
2. Similarities:
   • sunset: $\frac{(6)(5)+(8)(10)}{10\cdot 11.180}=\frac{110}{111.80}\approx \mathrm{0.9839.}$
   • car: $\frac{(6)(9)+(8)(1)}{10\cdot 9.055}=\frac{62}{90.55}\approx \mathrm{0.6847.}$
   • cat: $\frac{(6)(-2)+(8)(-1)}{10\cdot 2.236}=\frac{-20}{22.36}\approx -\mathrm{0.8944.}$ Why this step? Zero-shot retrieval = rank candidates by cosine; no classifier trained.
3. Softmax, $\tau =0.1$: scale to $9.839,6.847,-8.944$; exponentiate and normalize $\Rightarrow P(\text{sunset})\approx \mathrm{0.9524.}$

Verify: highest similarity is "sunset over mountains" ($0.9839$) — matches the forecast. The negative cat score confirms opposite-direction arrows are actively rejected. Product decision: return the sunset caption, confidence $\approx 95\%$. Units check: similarities dimensionless in $[-1,1]$ ✓, probabilities sum to $1$ ✓.

---

Example 8 — Cell J: exam twist (noisy pairs + symmetric loss)

Forecast: with only a $0.05$ gap, guess whether $P(\text{correct})$ is barely above $0.5$ or strongly above.

1. Scale: $0.55/0.1=5.5$, $0.50/0.1=\mathrm{5.0.}$
2. Exponentiate: $e^{5.5}\approx 244.69$, $e^{5.0}\approx \mathrm{148.41.}$
3. Normalize: $P(\text{correct})=244.69/(244.69+148.41)\approx \mathrm{0.6225.}$ Why? Even a small $0.05$ edge, magnified by $1/\tau =10$ and $\exp$, becomes a $62\%$ vs $38\%$ preference — the "signal accumulates" claim from the parent, made concrete.
4. Symmetry check: if the similarity matrix is symmetric ($s_{ij}=s_{ji}$), then image→text row $[0.55,0.50]$ and text→image column $[0.55,0.50]$ are the same numbers, so ${\mathcal{L}}_{\text{img→txt}}={\mathcal{L}}_{\text{txt→img}}$, and the averaged loss equals either one. Why the trap: students assume symmetry is automatic — it is not; the two losses coincide only when the score matrix is symmetric, which contrastive setups do not guarantee, so the average genuinely matters.

Verify: $P(\text{correct})\approx 0.6225>0.5$ ✓ — noisy but still correctly biased. Loss $=-\log (0.6225)\approx 0.4740$. This is the mathematical reason billions of noisy pairs beat millions of clean ones.

---

Recall Self-test (reveal after guessing)

What operation makes cosine similarity ignore an arrow's length? ::: Dividing the dot product by both magnitudes $\Vert u\Vert \Vert w\Vert$. A zero vector's cosine similarity is... ::: Undefined ($0/0$) — it has no direction. As $\tau \to 0$, softmax becomes... ::: A hard argmax (winner takes all). As $\tau \to \infty$, softmax becomes... ::: Uniform, probability $1/B$ each. Why does the loss average image→text and text→image? ::: Because the similarity matrix need not be symmetric, and both directions must be correct.

See also: 6.1.04-Word-embeddings, 6.5.06-Text-to-image-generation.