Question bank — Self-supervised and contrastive learning (SimCLR, CLIP)

False. It manufactures labels (pseudo-labels) from the data's own structure and then trains supervised-style on them; unsupervised methods (clustering, density) never form a label at all.

False. Positives are made for free — two augmentations of one image (SimCLR) or an image and its scraped caption (CLIP). No human labels the pair.

False — it's the opposite. Dividing similarities by a smaller $\tau$ blows up the differences, sharpening the softmax so the model is punished harder unless the positive clearly wins.

True. The loss is applied on $z=g(h)$, but $h$ is kept for transfer; the projection head absorbs information useful only for the pretext task, so $h$ transfers better empirically.

True. Dividing by $\Vert u\Vert \Vert v\Vert$ cancels magnitude, leaving only the angle — that is exactly why embeddings are usually L2-normalized before the loss.

False. For SimCLR the batch is the negative pool ($2N-2$ negatives), so batch size directly changes the learning signal; for standard supervised training batch size mostly affects optimization noise, not the objective.

False. CLIP is symmetric: image→text loss (softmax over each row of $S$) plus text→image loss (softmax over each column of $S$), averaged — two InfoNCE terms sharing one similarity matrix.

False. Zero-shot classification is built from matching: turn each class name into a caption ("a photo of a {dog}") and pick the caption with highest similarity. See Zero-shot and few-shot learning.

False. Without $g$, the contrastive loss acts directly on $h$, forcing $h$ to become invariant to augmentations it should retain info about — measured downstream accuracy drops.

True. Row $i$'s only positive is column $i$ (its true caption); every off-diagonal $S_{ij}$ ($j\ne i$) in that row is a negative.

Error. Look at the InfoNCE formula: the positive term appears in the denominator sum, and it must — the loss is the negative log-probability of picking the positive among all candidates, so the positive is one of the choices being normalized over.

Error. In the InfoNCE formula the indicator $1_{k\ne i}$ zeroes out exactly the $k=i$ term, so the anchor is never compared to itself; that self-comparison (sim $=1$) would swamp the sum and give a meaningless gradient.

Error. Those ten views are all positives of each other, not negatives. Negatives come from different source images in the batch; more augmentations of one image add positives, not the contrast you want.

Error. The text→image loss softmaxes down the columns of $S$; skipping it makes the objective asymmetric and lets the image side dominate the shared space.

Error. It ranges from $-1$ (opposite directions) to $+1$ (same direction); $0$ means orthogonal. Negatives can and should push similarity toward negative values, not merely toward zero.

Error. Even setting aside that $S_{ij}$ compares image $i$ to caption $j$ (not symmetric in general), softmax over row $i$ normalizes against different competitors than softmax over column $i$, so the two losses differ and both are needed.

Error. $\tau =0$ divides by zero; as $\tau \to 0$ the softmax collapses to a hard argmax, killing gradients for all but the single closest sample and destroying training. There is a useful minimum, not zero.

Error. Both directions are summed — each view of a pair serves as anchor once, so with the per-view loss ${\ell }_{i,j}$ defined above, both ${\ell }_{2k-1,2k}$ and ${\ell }_{2k,2k-1}$ appear in the total.

It turns the softmax probability into a log-probability, converting products into sums and giving numerically stable, well-scaled gradients — minimizing $-\log p$ is maximizing the probability of picking the positive.

Cosine strips out vector length so the model can't cheat by simply making all vectors huge; only the direction (semantic content) is compared, which is what we want to align.

Weak augmentations let the model match views by trivial low-level cues (same pixels, same color histogram). Strong augmentation destroys those shortcuts, forcing genuinely semantic, invariant features — see Data augmentation techniques.

To separate positives from many negatives, the encoder must capture object parts, shapes, and spatial relations — general features that reappear in classification/detection. That is the whole promise of Representation learning and Transfer learning and fine-tuning.

CLIP's supervision is language, so any new class expressed as text lands in the same shared space; a fixed softmax head only knows its trained label set. This is the Multi-modal learning payoff.

Every other sample in the batch is a negative, so a bigger batch means a richer, harder pool of negatives ($2N-2$ of them), giving a stronger contrastive signal per step.

It's specialized to the contrastive pretext task and may throw away info (like color) that hurts the pretext but helps downstream tasks; keeping $h$ before it preserves that richer representation.

The (image, alt-text) caption already is the supervision, harvested for free from the internet — the label problem is offloaded to whoever wrote the caption.

Small $\tau$ magnifies similarity gaps, so near-miss (hard) negatives dominate the gradient; large $\tau$ flattens the distribution so all negatives contribute roughly equally.

The denominator equals just the positive term, so the fraction is $1$ and the loss is $-\log 1=0$ — no gradient, no learning. Contrastive learning is meaningless without negatives.

They get treated as negatives and pushed apart even though they're semantically alike — a false negative. This is a known noise source; large batches make it statistically rare but never zero.

Then every similarity is $1$, positives and negatives are indistinguishable, and the loss is stuck high. Contrastive learning avoids this collapse precisely because negatives penalize making everything similar — unlike some non-contrastive methods that need extra tricks.

The off-diagonal for that duplicate is falsely labeled negative, so the model is pushed to separate a legitimately matching image–caption pair — mild label noise the objective silently tolerates at scale.

It is $-1$, the minimum possible. Negatives ideally drive toward here, but in practice orthogonality (sim $\approx 0$) is a more common and sufficient outcome.

All similarities get divided down toward zero, so $\exp$ of each is $\approx 1$ and the softmax becomes uniform — the positive gets no special weight and the model learns almost nothing.

The two "views" would be pixel-identical, so the model matches them by trivial memorization with no invariance learned — the representations wouldn't transfer.

The scale of the dot products drifts with vector magnitude, so $\tau$ no longer has a consistent meaning and one large-magnitude sample can dominate — this is why L2-normalization (making it true cosine similarity) is standard.