Multimodal models (vision-language): Exercises

This page is a self-testing ladder for Multimodal models (vision-language). Each problem has a hidden full solution. Try first, then reveal. Levels climb from recognising the pieces to synthesising a whole training loop.

Before we start, one picture to fix all the vocabulary in your head — everything below refers back to it.

Figure — Multimodal models (vision-language)

Level 1 — Recognition

L1.1

Match each term to its one-line job: (a) vision encoder, (b) text encoder, (c) joint embedding space, (d) cosine similarity.

Recall Solution

(a) Vision encoder — turns a grid of pixels into a feature vector. (See 6.5.03-VisionTransformers.)
(b) Text encoder — turns a sequence of word-tokens into a feature vector. (See 6.1.04-Word-embeddings.)
(c) Joint embedding space — the single $d$ -dimensional space both vectors are pushed into, so they can be compared.
(d) Cosine similarity — the number in $[- 1, 1]$ telling us how aligned two vectors' directions are.

L1.2

Cosine similarity of two vectors is $sim = + 1$ . What does that mean geometrically? What about $0$ and $- 1$ ?

Recall Solution

$+ 1$ ::: the arrows point in exactly the same direction (angle between them $= 0^{\circ}$ ).
$0$ ::: the arrows are perpendicular (angle $= 9 0^{\circ}$ ) — unrelated concepts.
$- 1$ ::: the arrows point exactly opposite (angle $= 18 0^{\circ}$ ). Cosine ignores length: only the angle matters.

L1.3

In CLIP training, a batch of $B$ pairs produces a similarity matrix. What is its shape, and what lives on the diagonal?

Recall Solution

Shape ::: $B \times B$ (every image scored against every text).
The diagonal entry $(i, i)$ is the score of image $i$ against its own correct caption — these are the ones we want largest.

Level 2 — Application

L2.1

Compute the cosine similarity of $v_{img} = (3, 4)$ and $v_{text} = (4, 3)$ .

Recall Solution

WHAT: plug into the formula $sim = \frac{v _{img} \cdot v _{text}}{∥ v _{img} ∥ ∥ v _{text} ∥} .$ Dot product (multiply matching entries, add): $3 \cdot 4 + 4 \cdot 3 = 12 + 12 = 24$ . Norms (length $= sum of squares$ ): $3^{2} + 4^{2} = 25 = 5$ for both. $sim = \frac{24}{5 \cdot 5} = \frac{24}{25} = 0.96.$ Very aligned — makes sense, the arrows are almost on top of each other.

L2.2

Same $v_{img} = (3, 4)$ , but now $v_{text} = (- 4, 3)$ . Compute the cosine. Interpret the sign.

Recall Solution

Dot: $3 \cdot (- 4) + 4 \cdot 3 = - 12 + 12 = 0$ . $sim = \frac{0}{5 \cdot 5} = 0.$ Zero means the arrows are perpendicular — the model treats these as unrelated concepts. Notice the sign of the dot product carried the whole story; the norms are always positive.

L2.3

A dog image scores against three text prompts with raw similarities $[0.85, 0.23, 0.05]$ for $[dog, cat, airplane]$ . Which class is predicted in zero-shot classification, and why does no training of a classifier occur?

Recall Solution

Predicted class: dog (highest similarity $0.85$ ). Why no classifier: the class names are just fed through the text encoder as ordinary text. We compare the image vector to each class-name vector and pick the closest. The alignment learned during contrastive pre-training does all the work — this is transfer via a shared space, not a trained output head.

Level 3 — Analysis

L3.1

For a batch of $4$ , the dog-image row of the similarity matrix is $[0.9, 0.1, 0.0, 0.1]$ (columns dog/cat/car/tree) with temperature $τ = 0.07$ . Compute the probability the model assigns to the correct pair, then the loss $L_{i} = - lo g P (correct)$ .

Recall Solution

WHY temperature: dividing each score by $τ = 0.07$ (a small number) blows the gaps up, making softmax sharp. Logits: $0.9/0.07 = 12.857$ , $0.1/0.07 = 1.4286$ , $0/0.07 = 0$ , $1.4286$ . Exponentiate: $e^{12.857} \approx 384, 000$ , $e^{1.4286} \approx 4.173$ , $e^{0} = 1$ , $4.173$ . $P = \frac{384000}{384000 + 4.173 + 1 + 4.173} \approx 0.99997.$ $L_{i} = - lo g (0.99997) \approx 2.4 \times 1 0^{- 5} .$ Tiny loss — the model is already confident and correct.

L3.2

Redo L3.1 with $τ = 1.0$ (no sharpening). Compare the correct-pair probability with the $τ = 0.07$ case and explain what temperature controls.

Recall Solution

Logits are just the raw similarities: $0.9, 0.1, 0.0, 0.1$ . Exponentiate: $e^{0.9} = 2.4596$ , $e^{0.1} = 1.1052$ , $e^{0} = 1$ , $1.1052$ . Sum $= 2.4596 + 1.1052 + 1 + 1.1052 = 5.6700$ . $P = \frac{2.4596}{5.6700} \approx 0.4338.$ With $τ = 1$ the correct probability drops to $\approx 0.43$ — the distribution is soft. Small $τ$ → sharp/confident; large $τ$ → soft/uncertain. That is why $τ$ is a learned knob controlling how hard the model must commit. See 5.3.03-Cross-entropyloss for the softmax underneath.

L3.3

The full CLIP loss is symmetric: $L = \frac{1}{2} (L_{img \to txt} + L_{txt \to img})$ . Explain why dropping one direction permits a degenerate solution.

Recall Solution

The image→text term only forces each image to pick its right caption among captions. A model could satisfy it by mapping every text vector to the same point and just spreading image vectors — image rows still classify correctly, but text is collapsed and useless. The text→image term punishes exactly this (each text must pick its right image among images). Requiring both directions removes the shortcut. This is a stability argument straight from 6.3.02-Self-supervised-learning.

Level 4 — Synthesis

L4.1

An image of side $224 \times 224$ is fed to a ViT with patch size $P = 16$ . How many patches $N$ ? Add one extra [CLS] token — what sequence length enters the transformer? (Background: 6.5.03-VisionTransformers.)

Recall Solution

$N = \frac{H \cdot W}{P ^{2}} = \frac{224 \cdot 224}{16 \cdot 16} = \frac{50176}{256} = 196.$ With the extra [CLS] token the sequence length is $196 + 1 = 197$ .

L4.2

You have $B = 4$ images and $4$ captions, embedding dimension $d = 512$ . How many entries are in the similarity matrix, and how many individual dot products must be computed to fill it (each dot product touching $2 d$ numbers)?

Recall Solution

Matrix entries: $B \times B = 4 \times 4 = 16$ . Each entry is one cosine → one dot product. So $16$ dot products. Numbers touched per dot product $= 2 d = 2 \cdot 512 = 1024$ ; total multiply-adds across the matrix $\approx 16 \cdot 512 = 8192$ multiplications (the $2 d$ counts reads, $d$ counts multiplies). Key insight: cost grows as $B^{2} d$ — the quadratic in batch size is why big contrastive batches are expensive.

L4.3

Design choice: Flamingo freezes its vision encoder and language model and only trains small cross-attention adapters. If the frozen parts hold $70, 000$ M parameters and the adapters hold $2, 000$ M, what fraction of parameters is actually trained? Why is this attractive?

Recall Solution

$fraction trained = \frac{2000}{70000 + 2000} = \frac{2000}{72000} \approx 0.0278 = 2.78%.$ Only $\approx 2.8%$ of weights update. Attractive because you reuse powerful pre-trained unimodal models (parameter-efficient 6.4.01-Transfer-learning), needing far less compute and data than training everything from scratch, while cross-attention injects the image information into the language stream.

Level 5 — Mastery

L5.1

Walk the entire contrastive step for a batch of $2$ : images $A, B$ with captions $a, b$ . Cosine matrix (rows = images, cols = texts) is $[0.8 0.3 0.2 0.7], τ = 0.1.$ Compute both per-image losses, both per-text losses, and the final symmetric loss $L$ .

Recall Solution

Scale by $1/ τ = 10$ : matrix of logits $[8327]$ .

Image→text (softmax over each ROW):

Row A: $e^{8} = 2980.96, e^{2} = 7.389$ ; $P_{A} = \frac{2980.96}{2988.35} = 0.99753$ ; $L_{A} = - lo g 0.99753 = 0.002475$ .
Row B: correct entry is $b$ (logit $7$ ): $e^{3} = 20.086, e^{7} = 1096.63$ ; $P_{B} = \frac{1096.63}{1116.72} = 0.98201$ ; $L_{B} = - lo g 0.98201 = 0.018155$ .
$L_{img \to txt} = \frac{1}{2} (0.002475 + 0.018155) = 0.010315$ .

Text→image (softmax over each COLUMN):

Col a: entries $e^{8} = 2980.96$ (image A, correct) and $e^{3} = 20.086$ ; $P_{a} = \frac{2980.96}{3001.05} = 0.993307$ ; $L_{a} = 0.006715$ .
Col b: entries $e^{2} = 7.389$ and $e^{7} = 1096.63$ (image B, correct); $P_{b} = \frac{1096.63}{1104.02} = 0.993306$ ; $L_{b} = 0.006716$ .
$L_{txt \to img} = \frac{1}{2} (0.006715 + 0.006716) = 0.0067155$ .

Final symmetric loss: $L = \frac{1}{2} (0.010315 + 0.0067155) \approx 0.008515.$ The picture below shows why the diagonal being largest drives the loss down.

L5.2

Argue why ALIGN's noisy web alt-text still trains a good model, using the contrastive signal directly.

Recall Solution

Contrastive loss only needs the correct pair to score slightly higher than the wrong pairs in the same batch. A noisy caption for image $i$ is still, on average, more relevant to image $i$ than to a random image $j$ . So the gradient nudges in the right direction on average. Individual mistakes add gradient noise but do not bias the direction; over $1.8$ B pairs the true signal accumulates and the noise averages out. This is the self-supervised payoff: scale beats curation because the objective is relative, not absolute.

L5.3

Suppose you double the batch size $B$ . Explain, using L3, why this generally makes the loss harder to drive down and improves the learned embeddings.

Recall Solution

Each image's softmax now runs over more competing texts (the denominator gains more $exp (sim / τ)$ terms). More competitors = more chances for a distractor to score high = higher, more informative loss = stronger gradients. The model must separate its correct pair from a larger crowd, forcing finer distinctions in the embedding space. That is precisely why CLIP used enormous batches — larger $B$ gives harder negatives for free.

Recall Quick self-check

Cosine of $(3, 4)$ and $(4, 3)$ ::: $0.96$ Patches for $224/16$ ViT (with CLS) ::: $196$ patches, sequence length $197$ Fraction of Flamingo params trained ::: $\approx 2.78%$ $τ$ small vs large ::: small = sharp/confident, large = soft/uncertain