4.5.16 · D5Generative Models
Question bank — Text-to-image conditioning (CLIP)
Before we start, the smallest possible vocabulary, so no symbol is unearned:
True or false — justify
Every item: state true/false and the reason. A bare verdict is worth nothing.
CLIP's cosine similarity depends on how "bright" or high-magnitude an embedding is.
False. Cosine divides by both arrow lengths , so only the direction survives — a doubled embedding gives the identical similarity.
If two embeddings are perpendicular, their cosine similarity is , meaning "maximally unrelated".
False. means unrelated, but the true opposite of "matching" is (arrows pointing against each other). Perpendicular is merely "no shared direction", the neutral middle.
Making the temperature smaller always improves CLIP training.
False. A tiny makes the logits huge and the softmax razor-sharp, so gradients explode or saturate; that is exactly why is learned and clamped rather than driven to zero.
CLIP needs human-written captions that exactly describe each image.
False. It trains on noisy web image–alt-text pairs; the contrastive signal only needs the matched pair to beat the other pairs in the batch, not a perfect caption. See 4.3.8-Contrastive-learning.
The symmetric CLIP loss is redundant — one direction already trains both encoders.
False. softmaxes over texts (each row of ); softmaxes over images (each column of ). Each direction only constrains one normalization axis, so both are needed to shape the full similarity matrix.
Increasing the guidance scale always produces a better image.
False. Larger pushes harder along , boosting prompt fidelity but crushing diversity and over-saturating colours; there is a sweet spot (typically –). See 4.5.15-Classifier-free-guidance.
At the classifier-free guidance formula reduces to plain conditional sampling.
True. Substituting gives — the difference term is fully applied exactly once, no amplification.
CLIP is a generative model that outputs images from text.
False. CLIP only scores how well an image and a text match. Generation is done by a separate diffusion model that CLIP merely steers via its text embedding.
Zero-shot classification with CLIP requires re-training on the new label set.
False. You just re-encode the new labels as text prompts and compare cosine similarities — no weight updates, which is the whole point of zero-shot transfer (4.2.12-Transfer-learning).
Spot the error
Each line contains a subtly wrong claim; the reveal names the flaw and fixes it.
"Cosine similarity can be any real number, so we scale by to bound it."
Backwards. is already bounded in ; dividing by widens the logits to to give the softmax something sharp to work with.
"The guidance term requires a separately trained classifier."
The error is in "separately trained". The classifier-free trick estimates the classifier gradient from the same network by randomly dropping the condition during training — no extra model.
"CLIP loss is cross-entropy against a uniform target over the batch."
The target is not uniform — it is one-hot: image must match text and no other, so the target puts all its probability on a single index, not spread flat.
"We use the dot product instead of cosine because it captures magnitude, which encodes confidence."
CLIP uses cosine (normalized dot product) precisely to discard magnitude; two same-meaning arrows must score high regardless of length, so raw dot product would leak irrelevant scale.
"With a batch of pairs there are negatives per image."
There are negatives per image (all texts except its own match); the th is the positive.
"Bayes' rule gives ."
The sign is wrong. It is a plus: , because turns the product into a sum.
"Because embeddings live in , the image and text encoders share weights."
Shared space is not shared weights. The two encoders are different networks (a vision transformer and a text transformer, see 4.4.7-Attention-mechanisms); they only agree on the output dimension .
Why questions
Why cosine similarity rather than Euclidean distance between embeddings?
We care about direction (meaning), not position; Euclidean distance would penalize embeddings for having different lengths even when perfectly aligned in meaning.
Why divide the logits by instead of multiplying?
Dividing by a small magnifies the logits so the softmax becomes confident; you tune "sharpness" by choosing how small is.
Why does the difference point "toward the prompt"?
predicts noise for the conditional distribution and for the marginal; their difference is exactly the score-gradient the condition adds, i.e. the direction that makes samples more prompt-like.
Why prepend "a photo of" to class labels in zero-shot classification?
CLIP's web captions rarely say a bare word like "dog"; the templated phrase lands nearer the training distribution, so its text embedding is more reliable.
Why is CLIP's contrastive objective preferred over supervised softmax over fixed classes?
Fixed classes lock you into a vocabulary; the contrastive space learns concepts, letting novel prompts like "steampunk octopus" be embedded and matched without ever being a training label.
Why must both and be normalized before computing similarity?
So the dot product literally equals cosine (which lives in ); without normalization a longer arrow would dominate the softmax regardless of alignment.
Why does classifier-free guidance drop the condition only – of the time, not ?
The model must mostly learn the strong conditional signal; a small dropout fraction is enough to also learn the unconditional needed for the difference term.
Edge cases
What is when ?
Exactly — identical arrows are perfectly aligned; this is the value CLIP training pushes the matching diagonal toward.
What happens to the softmax probabilities as ?
All logits shrink to , so the softmax becomes uniform ( each) — the model becomes maximally undecided and the loss stops distinguishing right from wrong pairs.
What happens as ?
The largest logit dominates completely, softmax collapses to a one-hot on the highest similarity — infinitely confident, and gradients vanish or blow up, which is why is never allowed near zero.
What does guidance scale produce?
: the prompt is entirely ignored, so you sample generic images from the unconditional distribution — no relation to the text.
Can two embeddings have cosine similarity below during a healthy training run?
Yes — negative pairs should drift toward (arrows pointing apart); only the diagonal positives are pushed toward .
What if the batch has size (a single pair)?
The softmax has one term, so and the cross-entropy is : with no negatives there is nothing to contrast against, so no learning signal exists — batch size must exceed one.
What if an image genuinely matches two texts in the batch equally (a near-duplicate caption)?
The one-hot target still credits only the paired text, so CLIP fights a correct-but-unlabeled positive as if it were a negative — a known label-noise cost of the batch-contrastive setup.
What does an image embedding with zero norm () do to cosine similarity?
It divides by zero — cosine is undefined; a degenerate (all-zero) embedding is a failure mode encoders avoid by never collapsing outputs to the origin.
Recall Two-line self test
Guidance formula collapses to plain conditional sampling at which ? ::: . Cosine similarity of two perpendicular embeddings? ::: , meaning unrelated, not opposite.
See also: 4.5.17-Multimodal-learning for where shared text–image spaces go next.