6.5.5 · D5Research Frontiers & Practice
Question bank — Multimodal models (vision-language)
This is a question bank for the parent topic. Each line is a trap: a place where intuition slips. Read the question, answer it in your head, then reveal. The answer side always gives you the reasoning, not just a verdict.
Prerequisites worth re-reading if a question stumps you: 6.2.05-Attention-mechanisms, 6.3.02-Self-supervised-learning, 5.3.03-Cross-entropyloss, 6.5.03-VisionTransformers, 6.1.04-Word-embeddings.
True or false — justify
TF1. "Cosine similarity of between an image vector and a text vector means they are the same vector."
False. Cosine measures direction only; two vectors of different lengths (say and ) both point the same way and score . Equal direction, not equal vector.
TF2. "A contrastive model needs explicitly labelled 'wrong' pairs to learn."
False. The negatives come for free: every other caption in the same batch is treated as a negative for a given image. No human labels the negatives — the batch structure supplies them.
TF3. "Making the temperature larger makes the model more confident."
False. divides the similarities before softmax, so a larger shrinks the gaps and gives a softer, less confident distribution. Smaller sharpens it.
TF4. "Because CLIP was never trained with a 'zebra' classifier head, it cannot classify zebras."
False. Zero-shot classification needs no head — you encode the word "zebra" as text and compare. If "zebra" appeared in captions during pre-training, its text embedding is meaningful and the image can match it.
TF5. "The image encoder and text encoder must have the same internal architecture so their outputs are comparable."
False. They can be totally different (ResNet vs. Transformer). Comparability comes from the final projection layers mapping both into one -dimensional space — not from matching internals.
TF6. "Contrastive loss trained only image→text is enough; the symmetric text→image term is redundant."
False. A one-sided loss admits degenerate solutions where every image maps near one text vector. The symmetric term forces the relation to hold both ways, which is what "alignment" actually means.
TF7. "ALIGN's noisy web alt-text should make it strictly worse than a hand-curated dataset of the same size."
Roughly false in practice. Contrastive loss only needs correct pairs to score slightly higher than wrong ones; over billions of examples the true signal accumulates and averages out the noise, so scale can beat curation.
TF8. "Positional encodings matter for the text encoder but not for the vision encoder."
False. A ViT flattens patches into a set with no inherent order, so it needs positional embeddings to know where each patch sits — exactly as text needs them for word order (see 6.5.03-VisionTransformers).
Spot the error
SE1. "To compare image and text, we take the dot product and call it the similarity."
The raw dot product mixes in magnitude; a longer vector scores higher regardless of direction. The fix is to divide by both norms — that is precisely why we use cosine, not the bare dot product.
SE2. "The softmax in InfoNCE runs over the two modalities (image vs. text)."
No — the softmax runs over the candidate texts (or images) in the batch. It is a -way classification asking "which caption is the true partner of this image?"
SE3. "In Flamingo we fine-tune the 70B language model on image-text data."
The LM and vision encoder are frozen. Only the small adapter pieces — the Perceiver Resampler and inserted cross-attention layers — are trained, which is what makes it parameter-efficient.
SE4. "Zero-shot works because the model memorised every test image during training."
It works through semantic generalisation, not memorisation. An unseen dog photo lands near the "dog" text because training pulled the whole region of dog-like images toward dog-like text (transfer, see 6.4.01-Transfer-learning).
SE5. "We split the image into patches so the model can classify each patch separately."
Patches are split so the transformer can attend across them and model long-range relations (tail-to-head), not to classify them independently.
SE6. "Cross-entropy and InfoNCE are unrelated losses."
InfoNCE is a cross-entropy loss — the softmax-then- of the correct class, where the "classes" are the batch candidates (see 5.3.03-Cross-entropyloss).
SE7. "A cosine similarity of between image and text means they are opposites."
means orthogonal / unrelated, not opposite. Opposite (contradictory direction) would be .
Why questions
WHY1. "Why not just concatenate pixels and word vectors into one big input to a single network?"
Pixels are a continuous 2D grid; tokens are discrete 1D symbols. They need different inductive biases (spatial vs. sequential), so separate encoders extract the right structure before meeting in the shared space.
WHY2. "Why is the temperature learned rather than fixed?"
The optimal sharpness of the distribution depends on how separable the data is; making a parameter lets the model discover the confidence level that minimises loss on its specific dataset.
WHY3. "Why does contrastive learning count as self-supervised?"
The supervision signal — "this image goes with this caption" — comes from data pairing already present on the web, with no human annotating labels. That is exactly the definition of 6.3.02-Self-supervised-learning.
WHY4. "Why use cosine similarity instead of Euclidean distance in the joint space?"
We care whether two concepts point the same direction, independent of vector length. Cosine ignores magnitude; Euclidean distance would wrongly penalise a correct match just for having a larger norm.
WHY5. "Why do bigger batches usually help contrastive training?"
A larger batch supplies more negatives per positive, so the softmax has more distractors to push apart — a harder, more informative task that sharpens the embeddings.
WHY6. "Why insert cross-attention into the LM in Flamingo instead of feeding image tokens at the start?"
Cross-attention lets every layer of the frozen LM query the image on demand while keeping the LM's own weights untouched — a clean, low-parameter injection point (see 6.2.05-Attention-mechanisms).
WHY7. "Why does the text 'a photo of a dog' often work better as a class prompt than just 'dog'?"
Captions in training were natural sentences, so a sentence-shaped prompt lands in a more familiar region of the text-embedding space, closer to how real captions were written.
Edge cases
EC1. "What happens if a batch accidentally contains two near-identical captions (e.g. two 'a cat' images)?"
The true partner and a false negative become almost equally similar, so the loss punishes the model for a match that should be correct. These "false negatives" add noise; huge diverse batches make them rare.
EC2. "What if an image contains a concept that never appeared in any caption during training?"
There is no text direction to align it with, so it lands somewhere arbitrary and zero-shot matching to that concept fails. The model can only recognise what language described to it.
EC3. "What does a similarity matrix look like at the very start of training (random weights)?"
All entries hover near (random directions in high dimensions are nearly orthogonal), so every softmax is near-uniform and the loss is close to — the maximum. Training then carves out the diagonal.
EC4. "Two different images with the same caption 'a red car' — must they get identical embeddings?"
No. They only need to both sit closer to the "a red car" text than to unrelated captions; the space can still separate a hatchback from a sedan while keeping both near that text.
EC5. "If , what happens to the loss surface?"
The similarities get divided by a tiny number, so softmax becomes an (almost) hard argmax — gradients vanish everywhere except at the top scorer, making training brittle and prone to collapse.
EC6. "A caption is empty text (just the sentence-start token). What does the text encoder output?"
A fixed, content-free vector pointing in some default direction; it aligns with nothing in particular, so any image paired with it gives a near-uniform, uninformative row.
EC7. "Can cosine similarity ever be undefined?"
Yes — if either vector is the zero vector, its norm is and you divide by zero. Real encoders avoid this because learned projections almost never output exactly zero, but it is the true degenerate boundary.
Recall Fast self-test
One-word triggers — say the full reason. Larger means... ::: softer, less confident distribution. Negatives in a contrastive batch come from... ::: the other pairs in that same batch. Cosine ignores... ::: magnitude, keeping only direction. Flamingo trains only... ::: the adapter layers (Resampler + cross-attention); the rest is frozen.