Self-supervised and contrastive learning (SimCLR, CLIP): Exercises

A quick reminder of the two tools we will use over and over, stated in plain words:

Classify each training setup as supervised, unsupervised, or self-supervised: (a) A model predicts the rotation angle applied to an unlabelled image. (b) A model is trained on images each tagged by a human with "cat" or "dog". (c) A model groups 10,000 unlabelled images into clusters, no target given.

Recall Solution 1.1

(a) Self-supervised — the rotation angle is a pseudo-label the data generated itself (a pretext task). (b) Supervised — human labels. (c) Unsupervised — finds structure with no labels or pseudo-labels. See Supervised learning fundamentals and Representation learning.

Problem 1.2 (L1)

In SimCLR, a batch has $N = 8$ real images. How many augmented views are created, and how many negatives does each view see?

Recall Solution 1.2

Each image is augmented twice, so views $= 2 N = 16$ . For any anchor view, everything except itself and its one positive partner is a negative: $2 N - 2 = 16 - 2 = 14 negatives .$

Level 2 — Application

Problem 2.1 (L2)

Two normalized embeddings point as $z_{a} = (1, 0)$ and $z_{b} = (0, 1)$ (both length 1). Compute $sim (z_{a}, z_{b})$ and describe the geometry.

Recall Solution 2.1

Dot product $= 1 \cdot 0 + 0 \cdot 1 = 0$ . Lengths are both $1$ , so $sim = 0/ (1 \cdot 1) = 0$ . Geometrically the arrows are perpendicular (90°) — see the right-angle pair in the figure above. Zero similarity = "the model considers these unrelated." Related idea: Metric learning and embedings.

Problem 2.2 (L2)

An anchor has cosine similarity $0.9$ with its positive and $0.2$ with each of three negatives. With temperature $τ = 0.5$ , compute the InfoNCE loss $ℓ$ .

Recall Solution 2.2

Divide each similarity by $τ = 0.5$ (i.e. multiply by 2): positive logit $= 1.8$ , each negative logit $= 0.4$ . $numerator = e^{1.8} = 6.0496$ $denominator = e^{1.8} + 3 e^{0.4} = 6.0496 + 3 (1.4918) = 10.525$ $ℓ = - lo g \frac{6.0496}{10.525} = - lo g (0.5748) = 0.5537$ (natural log). A loss near $0.55$ means the model already assigns $\sim 57%$ probability to the right partner.

Problem 2.3 (L2)

Recompute Problem 2.2 but with $τ = 0.1$ . Does the loss go up or down? Explain in one line.

Recall Solution 2.3

Divide by $τ = 0.1$ (multiply by 10): positive logit $= 9$ , negatives $= 2$ each. $ℓ = - lo g \frac{e ^{9}}{e ^{9} + 3 e ^{2}} = - lo g \frac{8103.08}{8103.08 + 22.167} = - lo g (0.99727) = 0.00273$ Loss dropped sharply. Smaller $τ$ sharpens the softmax, so a modest similarity gap ( $0.9$ vs $0.2$ ) becomes a huge probability gap — the model looks far more confident.

Level 3 — Analysis

Problem 3.1 (L3)

Two embeddings are not normalized: $u = (3, 4)$ , $v = (6, 8)$ . Show that raw dot product and cosine similarity tell different stories, and say which one contrastive learning uses and why.

Recall Solution 3.1

Dot product $= 3 \cdot 6 + 4 \cdot 8 = 18 + 32 = 50$ — large, but that mostly reflects lengths ( $∥ u ∥ = 5$ , $∥ v ∥ = 10$ ). $sim = \frac{50}{5 \cdot 10} = \frac{50}{50} = 1.0$ Cosine similarity is exactly $1$ : the arrows point the same direction ( $v = 2 u$ ). Contrastive learning uses cosine so that "meaning = direction," not magnitude — otherwise a model could cheat by simply scaling embeddings longer to inflate similarity. See Representation learning.

Problem 3.2 (L3)

In SimCLR with batch $N = 4096$ , each sample contrasts against how many negatives? Why does the paper push batch size so high, and what breaks if the batch is tiny (say $N = 2$ )?

Recall Solution 3.2

Negatives $= 2 N - 2 = 8190$ . Large batches supply a rich, diverse set of negatives, so the softmax denominator samples many hard comparisons — a strong learning signal. With $N = 2$ you get only $2 N - 2 = 2$ negatives; the task "pick the right partner out of 3" is nearly trivial, gradients are weak, and representations collapse toward being unable to separate classes. This is why SimCLR needs 4096–8192 (and large-batch tricks from Batch normalization and training techniques).

Problem 3.3 (L3)

Given the similarity matrix $S$ below (rows = images, columns = captions), CLIP wants the diagonal to dominate. Compute the image-to-text loss for row 1 with $τ = 1$ . $S = 200020002$

Recall Solution 3.3

Row 1 logits: $S_{11} = 2$ , $S_{12} = 0$ , $S_{13} = 0$ . With $τ = 1$ : $L_{i 2 t}^{(1)} = - lo g \frac{e ^{2}}{e ^{2} + e ^{0} + e ^{0}} = - lo g \frac{7.389}{7.389 + 1 + 1} = - lo g (0.7869) = 0.2397$ The diagonal being higher than off-diagonals gives a low per-row loss — exactly what CLIP's row-wise softmax rewards. See Multi-modal learning.

Level 4 — Synthesis

Problem 4.1 (L4)

Design the CLIP total loss for a batch of $N = 2$ with similarity matrix (temperature $τ = 1$ ): $S = (3113)$ Compute $L_{i 2 t}$ , $L_{t 2 i}$ , and $L_{CLIP}$ .

Recall Solution 4.1

Image-to-text (softmax across each row): Row 1: $- lo g \frac{e ^{3}}{e ^{3} + e ^{1}} = - lo g \frac{20.0855}{20.0855 + 2.7183} = - lo g (0.8808) = 0.1269$ . Row 2: same by symmetry $= 0.1269$ . Average $L_{i 2 t} = 0.1269$ .

Text-to-image (softmax down each column): $S$ is symmetric, so columns match rows and $L_{t 2 i} = 0.1269$ .

$L_{CLIP} = \frac{1}{2} (0.1269 + 0.1269) = 0.1269 .$ Both directions agree because $S$ is symmetric — a clean illustration of the "symmetric dance."

Problem 4.2 (L4)

You have image $x$ . Sketch (in words + one formula chain) the full SimCLR forward path for one view, from raw image to the vector on which loss is computed. Name every function and space, and explain why the projection head exists.

Recall Solution 4.2

Chain: augment $\to$ encode $\to$ project. $\tilde{x} = t (x) f h = f (\tilde{x}) \in R^{d} g z = g (h) = W^{(2)} σ (W^{(1)} h) \in R^{128}$

$t$ : a random augmentation drawn from crop/color/blur (Data augmentation techniques).
$f$ : base encoder (ResNet or Vision transformers (ViT)) giving representation $h$ .
$g$ : small MLP projection head; loss is applied to $z$ , but $h$ is kept for downstream tasks (Transfer learning and fine-tuning).
Why the head? Empirically $h$ transfers better than $z$ . The head can absorb information useful only for the contrastive pretext task (e.g. invariance to exact color jitter) so that $h$ stays richer for classification. Loss on $z$ , transfer on $h$ .

Problem 4.3 (L4)

Explain how CLIP does zero-shot classification of a new image into classes {cat, dog, car} without any training on those labels. Give the exact scoring rule.

Recall Solution 4.3

Build a caption per class: "a photo of a cat", etc. Encode each with the text encoder $T$ to get ${T_{cat}, T_{dog}, T_{car}}$ . Encode the image with $I$ . Predict: $\overset{y}{^} = ar g max_{c} sim (I (x), T_{c}) .$ The class whose caption embedding is closest in direction to the image embedding wins — no gradient step needed. This is exactly Zero-shot and few-shot learning powered by the shared embedding space.

Level 5 — Mastery

Problem 5.1 (L5)

Edge case — the collapse. Suppose an encoder maps every input to the same vector $z = (1, 0)$ . Show what the InfoNCE loss becomes with one positive and $M$ negatives, and explain why the temperature/normalization does not save you, and what actually prevents collapse.

Recall Solution 5.1

If all embeddings are identical, every similarity is $sim = 1$ (each vector with itself, normalized). Every logit equals $1/ τ$ , so numerator and each of the $M + 1$ denominator terms are all $e^{1/ τ}$ : $ℓ = - lo g \frac{e ^{1/ τ}}{( M + 1 ) e ^{1/ τ}} = - lo g \frac{1}{M + 1} = lo g (M + 1) .$ This is a constant floor — for $M = 14$ , $ℓ = lo g 15 = 2.708$ . The loss is not zero, so gradients still exist pushing away from collapse; but the positive term gets no advantage over negatives, so a naive optimizer can sit near this plateau. What genuinely prevents collapse is that positives (augmented views) share content negatives don't, so a non-collapsed solution achieves loss below $lo g (M + 1)$ . Strong augmentation + many diverse negatives make that lower solution reachable.

Problem 5.2 (L5)

Limiting behaviour of $τ$ . Take one positive with similarity $s^{+} = 0.8$ and one negative with $s^{-} = 0.6$ . Find $lim_{τ \to 0^{+}} ℓ$ and $lim_{τ \to \infty} ℓ$ . Interpret each limit.

Recall Solution 5.2

$ℓ (τ) = - lo g \frac{e ^{0.8/ τ}}{e ^{0.8/ τ} + e ^{0.6/ τ}} = - lo g \frac{1}{1 + e ^{(0.6 - 0.8) / τ}} = lo g (1 + e^{- 0.2/ τ}) .$

$τ \to 0^{+}$ : $- 0.2/ τ \to - \infty$ , so $e^{- 0.2/ τ} \to 0$ and $ℓ \to lo g (1) = 0$ . Tiny temperature makes the softmax infinitely sharp: any positive-negative gap becomes "perfect confidence," loss vanishes. Danger: overconfident, sensitive to noise.
$τ \to \infty$ : $- 0.2/ τ \to 0$ , so $e^{0} = 1$ and $ℓ \to lo g (2) = 0.6931$ . Huge temperature flattens the softmax to a uniform guess — the model can't tell positive from negative, maximal confusion (for one negative, chance $= 1/2$ , loss $= lo g 2$ ). This is why practical $τ \in [0.1, 0.5]$ : sharp enough to learn, not so sharp it overfits noise.

Problem 5.3 (L5)

Full worked InfoNCE with the parent's numbers. From the parent's SimCLR example: anchor cat₁ has positive cat₂ with $s = 0.85$ , and six negatives with similarities ${0.2, 0.15, 0.3, 0.1, 0.25, 0.2}$ , $τ = 0.5$ . Compute $ℓ$ exactly.

Recall Solution 5.3

Multiply every similarity by $1/ τ = 2$ : positive logit $= 1.7$ ; negative logits $= {0.4, 0.3, 0.6, 0.2, 0.5, 0.4}$ . $num = e^{1.7} = 5.4739$ $neg sum = e^{0.4} + e^{0.3} + e^{0.6} + e^{0.2} + e^{0.5} + e^{0.4} = 1.4918 + 1.3499 + 1.8221 + 1.2214 + 1.6487 + 1.4918 = 9.0257$ $den = 5.4739 + 9.0257 = 14.4996$ $ℓ = - lo g \frac{5.4739}{14.4996} = - lo g (0.37752) = 0.9741$ This matches the parent note's " $\approx 1.01$ " estimate (it rounded the negative sum). Exact value $\approx 0.974$ .

Recall Self-test recap

Cosine similarity ignores what and keeps what? ::: Ignores magnitude, keeps direction (angle). Where does the positive term appear in InfoNCE? ::: In BOTH numerator and denominator. Negatives per SimCLR anchor for batch $N$ ? ::: $2 N - 2$ . Which vector transfers downstream in SimCLR, $h$ or $z$ ? ::: $h$ (before the projection head). As $τ \to 0^{+}$ with a positive-negative gap, the loss tends to? ::: $0$ (infinitely sharp softmax). CLIP's zero-shot rule? ::: Pick the class whose caption embedding has max cosine similarity to the image embedding.

Exercises — Self-supervised and contrastive learning (SimCLR, CLIP)

Level 1 — Recognition

Problem 1.1 (L1)