Self-supervised and contrastive learning (SimCLR, CLIP): Worked examples

This page is the "plug-in-the-numbers" companion to the parent topic. There we built the ideas: positive pairs pulled together, negative pairs pushed apart, the InfoNCE loss that scores it all. Here we grind through every kind of situation that loss can face — every sign of similarity, the perfect and the disastrous cases, the degenerate zero-negative case, the temperature limits, a real-world word problem, and an exam twist.

Before we start, one promise: we define every symbol on its first appearance, even ones the parent used. If a line uses a letter you have not met on this page, that is a bug.

Ground vocabulary (earn every symbol)

Definition The symbols we will reuse everywhere

$z$ — a representation vector, a list of numbers a network spits out for one input (image or text). Picture an arrow pointing out from the origin in a flat space.
$z^{+}$ — the positive: the representation of a matching item (a second augmented view of the same image, or the correct caption for an image). It is the arrow we want our anchor $z$ to align with.
$z_{i}^{-}$ — the $i$ -th negative: the representation of a non-matching item (a different image, a wrong caption). The subscript $i$ just numbers them $1, 2, 3, \dots$ ; these are the arrows we want $z$ to point away from.
$N$ — the number of negatives for one anchor (a plain counting integer). So one anchor faces $N + 1$ candidates in total: its single positive plus $N$ negatives.
$sim (u, v)$ — cosine similarity between two arrows $u$ and $v$ . It is $\frac{u ^{⊤} v}{∥ u ∥ ∥ v ∥}$ , which is just $cos (angle between them)$ . So it lives in $[- 1, + 1]$ : $+ 1$ means arrows point the same way, $0$ means perpendicular, $- 1$ means opposite.
$τ$ (tau) — the temperature, a positive knob. We divide every similarity by $τ$ . Small $τ$ magnifies differences; large $τ$ flattens them. Why divide and not multiply? Because dividing by a small number blows the gap up — that is the "sharpening" we want.
$ℓ$ — the loss for one anchor: a single non-negative number. $0$ = perfect, larger = worse.

Two more words we will lean on in every example — logit and softmax — defined now so no example uses them cold:

The one-anchor InfoNCE loss we score all page long — every symbol in it ( $z$ , $z^{+}$ , $z_{i}^{-}$ , $N$ , $sim$ , $τ$ ) was just defined above:

To keep the tables and examples short, we abbreviate the two similarities we care about:

The scenario matrix

Every situation contrastive loss can face reduces to a combination of: how similar the positive is ( $s^{+}$ ), how similar the worst negative is ( $s^{-}$ ), how many negatives there are ( $N$ ), and how sharp $τ$ is. Here is the full grid — each example below is tagged with the cell it hits. In the " $τ$ / #negatives" column, #negatives means $N$ , the count defined above.

Cell	Positive sim $s^{+}$	Negative sims $s^{-}$	$τ$ / #negatives $N$	What it tests
A Ideal	high ( $\approx 1$ )	low ( $\approx 0$ )	normal $τ$ , $N = 3$	best-case loss near 0
B Confused	medium	one negative higher than positive	normal $τ$ , $N = 2$	loss when a distractor wins
C Negative sim	positive $> 0$	a negative with $s^{-} < 0$ (opposite arrows)	normal $τ$ , $N = 2$	the full $[- 1, 1]$ range
D Degenerate	any	zero negatives ( $N = 0$ )	any $τ$ , $N = 0$	what happens with no contrast
E $τ \to 0$	fixed gap	fixed	$τ$ tiny, $N = 1$	hard-negative limit
F $τ \to \infty$	fixed gap	fixed	$τ$ huge, $N = 1$	soft / uniform limit
G Word problem	—	—	CLIP zero-shot	pick a class from captions
H Exam twist	symmetric matrix	—	CLIP i2t + t2i	full batch loss both directions

Cells A, B, C, D, E, F, G, H are each covered below.

Worked examples

Steps:

Divide every similarity by $τ = 0.5$ (i.e. multiply by $2$ ). Why this step? $τ$ sets the scale before the exponential; skipping it changes the answer entirely. We get logits $1.80$ (positive) and $0.20, 0.10, 0.30$ (negatives).
Exponentiate each logit. $e^{1.80} = 6.0496$ ; $e^{0.20} = 1.2214$ ; $e^{0.10} = 1.1052$ ; $e^{0.30} = 1.3499$ . Why? $exp$ turns similarities into positive "scores" so the softmax fraction reads as a probability.
Form the probability of the positive. Numerator $= 6.0496$ ; denominator $= 6.0496 + 1.2214 + 1.1052 + 1.3499 = 9.7261$ . Probability $= 0.6220$ . Why? This fraction is the softmax probability the model assigns to "the positive is the match."
Take $- lo g$ . $ℓ = - ln (0.6220) = 0.4749$ . Why? Loss is the surprise of the correct choice; low surprise = low loss.

Verify: probability $0.622 \in (0, 1]$ ✓; loss $0.475 > 0$ and modest ✓. Sanity: a perfect $p = 1$ would give $ℓ = 0$ ; we are close-ish, consistent with a strong but not overwhelming lead.

Steps:

Scale by $1/ τ = 2$ : logits $0.80$ (positive), $1.40$ , $0.20$ (negatives). Why? Same reason — temperature first.
Exponentiate: $e^{0.80} = 2.2255$ , $e^{1.40} = 4.0552$ , $e^{0.20} = 1.2214$ . Why? Convert to scores.
Positive probability: $\frac{2.2255}{2.2255 + 4.0552 + 1.2214} = \frac{2.2255}{7.5021} = 0.2967$ . Why? The hard negative $4.0552$ dominates the denominator, stealing probability mass.
$- lo g$ : $ℓ = - ln (0.2967) = 1.2148$ . Why? High surprise → high loss → big gradient pushing that distractor away.

Verify: $ℓ = 1.215 > 0.475$ from Example 1 ✓ — being fooled costs more, exactly as it should. Probability $0.297 < 0.5$ means the model would not pick the positive if forced to guess ✓.

Steps:

Scale by $2$ : logits $1.20$ , $- 1.60$ , $0.00$ . Why this step? Note a negative similarity gives a negative logit — $exp$ of it will be tiny, contributing almost nothing.
Exponentiate: $e^{1.20} = 3.3201$ , $e^{- 1.60} = 0.2019$ , $e^{0} = 1.0000$ . Why? See how the opposite-pointing negative ( $0.2019$ ) barely matters — that is cosine's full range doing useful work.
Positive probability: $\frac{3.3201}{3.3201 + 0.2019 + 1.0000} = \frac{3.3201}{4.5220} = 0.7342$ . Why? This fraction is the InfoNCE numerator over its denominator — the softmax probability the model assigns to the positive being the match, exactly the fraction inside the $- lo g$ of the loss formula.
$- lo g$ : $ℓ = - ln (0.7342) = 0.3090$ . Why? The InfoNCE loss is $- lo g$ of that probability; taking the negative log converts "high probability of the right choice" into "low loss," completing the formula.

Verify: $ℓ = 0.309 < 0.475$ (Example 1) ✓ — having a strongly repelled negative helps, because it barely competes. Cosine handled $- 0.8$ without any special case ✓.

Figure — Cell C geometry. The picture below draws the anchor $z$ (navy) and the three candidate arrows: the positive $z^{+}$ at cosine $0.60$ (violet), the perpendicular negative $z_{2}^{-}$ at cosine $0.00$ (orange), and the nearly-opposite negative $z_{1}^{-}$ at cosine $- 0.80$ (magenta). Beside each arrow is its $exp (sim / τ)$ contribution — watch the magenta arrow's contribution collapse to $0.20$ , which is why an opposite-pointing negative barely competes in the denominator.

Figure — Self-supervised and contrastive learning (SimCLR, CLIP)

Steps:

Write the fraction with an empty sum. The denominator is $exp (s^{+} / τ) + \sum_{empty} = exp (s^{+} / τ) + 0$ . Why this step? An empty sum equals $0$ ; the denominator collapses to just the numerator.
Simplify. Numerator $=$ denominator $= exp (s^{+} / τ)$ , so the fraction $= 1$ . Why? There is nothing to steal probability, so the positive gets all of it.
$- lo g (1) = 0$ . Why? Zero surprise. But learning stalls: the gradient is $0$ , the encoder can output anything.

Verify: independent of $s^{+}$ and $τ$ , $ℓ = 0$ ✓. Mistake alert below — this is why SimCLR needs large batches: with $N = 0$ negatives the loss is uninformative.

Steps:

Write the probability. $p = \frac{e ^{0.5/ τ}}{e ^{0.5/ τ} + e ^{0.3/ τ}} = \frac{1}{1 + e ^{(0.3 - 0.5) / τ}} = \frac{1}{1 + e ^{- 0.2/ τ}}$ . Why this step? Dividing top and bottom by $e^{0.5/ τ}$ isolates the gap $0.3 - 0.5 = - 0.2$ .
Take $τ \to 0^{+}$ . Then $- 0.2/ τ \to - \infty$ , so $e^{- 0.2/ τ} \to 0$ , giving $p \to 1$ . Why? A positive gap divided by a vanishing $τ$ explodes; the exponential of $- \infty$ vanishes.
So $ℓ \to - lo g 1 = 0$ . Why? With infinite sharpening, any lead of the positive becomes a certain win.

Numeric taste at $τ = 0.05$ : $p = 1/ (1 + e^{- 4}) = 0.98201$ , $ℓ = 0.01815$ .

Verify: as $τ$ shrinks $ℓ$ drops toward $0$ ✓. Caveat: this only helps if the positive actually leads; if a negative led, the same sharpening would drive loss to $+ \infty$ (the danger of small $τ$ ).

Steps:

Use $p = \frac{1}{1 + e ^{- 0.2/ τ}}$ from Example 5. Why? Same algebra, different limit.
Take $τ \to \infty$ . Then $- 0.2/ τ \to 0$ , $e^{0} = 1$ , so $p \to \frac{1}{1 + 1} = \frac{1}{2}$ . Why? Infinite temperature erases the similarity gap; all candidates look equal.
$ℓ \to - lo g (1/2) = 0.6931$ . Why? With $2$ equally likely candidates, the loss is the entropy of a fair coin.

Numeric taste at $τ = 100$ : $p = 1/ (1 + e^{- 0.002}) = 0.50050$ , $ℓ = - ln (0.50050) = 0.69215$ .

Verify: $ℓ \to 0.6931 = ln 2$ ✓. General rule: as $τ \to \infty$ with $M$ total candidates, $ℓ \to lo g M$ , where $M = N + 1$ is the total number of candidates (positive plus $N$ negatives). Here $M = 2$ .

Steps:

Scale by $1/ τ = 100$ : logits $28, 22, 5$ . Why this step? CLIP's tiny temperature turns small cosine gaps into large logit gaps — that is how it makes crisp decisions.
Exponentiate each logit (subtract the max $28$ first for numerical stability): $e^{0} = 1$ , $e^{- 6} = 0.0024788$ , $e^{- 23} \approx 1.03 \times 1 0^{- 10}$ . Why subtract the max? Softmax is unchanged if we shift all logits by a constant (the shift cancels top and bottom), but it prevents $e^{28}$ from overflowing.
Form the probability of the cat caption (softmax over the three captions): $p_{cat} = \frac{1}{1 + 0.0024788 + 1.0 \times 1 0 ^{- 10}} = 0.99753$ . Why? This is InfoNCE with the correct caption as the positive and the other two captions as negatives — the softmax probability is the zero-shot classification confidence.
Take $- lo g$ for the loss / read off the answer. Predict cat; the associated loss would be $- ln (0.99753) = 0.002472$ . Why? The $- lo g$ of the positive's probability is the InfoNCE loss; a near- $1$ probability means near- $0$ loss and a confident, correct prediction.

Verify: probabilities sum to $1$ ✓; cat's $0.06$ cosine lead $\times 100 = 6$ logit gap gives $e^{6} \approx 403$ -to- $1$ odds over dog, matching $0.99753/0.002472$ ✓. This is exactly transfer with no fine-tuning — pure zero-shot.

Steps:

Image→text: softmax each ROW, keep the diagonal. Row 1: $p_{11} = \frac{e ^{4}}{e ^{4} + e ^{1}} = \frac{54.598}{54.598 + 2.718} = 0.95257$ . Row 2: $p_{22} = \frac{e ^{3}}{e ^{0} + e ^{3}} = \frac{20.086}{1 + 20.086} = 0.95257$ . Why rows? Image-to-text asks "for this image, which caption?" — that is a softmax distribution across captions = across the row, with the diagonal entry as the positive logit.
$L_{i 2 t} = - \frac{1}{2} (ln 0.95257 + ln 0.95257) = - ln 0.95257 = 0.04859$ . Why average? Each image contributes one InfoNCE term ( $- lo g$ of its positive's probability); we mean over the $N = 2$ images.
Text→image: softmax each COLUMN, keep the diagonal. Col 1: $p_{11} = \frac{e ^{4}}{e ^{4} + e ^{0}} = \frac{54.598}{54.598 + 1} = 0.98201$ . Col 2: $p_{22} = \frac{e ^{3}}{e ^{1} + e ^{3}} = \frac{20.086}{2.718 + 20.086} = 0.88079$ . Why columns? Text-to-image asks "for this caption, which image?" — a softmax distribution down the column, again with the diagonal as positive. 3.5. (same numbering intent)
$L_{t 2 i} = - \frac{1}{2} (ln 0.98201 + ln 0.88079) = - \frac{1}{2} (- 0.01816 - 0.12693) = 0.07254$ . Why? Average the two column-wise InfoNCE surprises, one per caption.
Total: $L_{CLIP} = \frac{1}{2} (L_{i 2 t} + L_{t 2 i}) = \frac{1}{2} (0.04859 + 0.07254) = 0.06057$ . Why symmetric? Both directions give gradient, so neither modality is ignored (see parent note).

Verify: $L_{i 2 t} = 0.0486$ , $L_{t 2 i} = 0.0725$ , total $0.0606$ ✓, inside the forecast band. Note $L_{i 2 t} \neq = L_{t 2 i}$ because rows and columns of $S$ differ — a healthy check that the two directions really are distinct computations.

Recall

Recall What does an empty negative set do to InfoNCE loss?

Makes it exactly $0$ regardless of $τ$ and $s^{+}$ — no learning signal (Cell D). ::: Loss $= 0$ , gradient $= 0$ , encoder unconstrained — a bug, not success.

Recall As

$τ \to \infty$ with $M = N + 1$ total candidates, the loss approaches? $lo g M$ ::: $lo g M$ — the uniform "everyone equal" loss (Cell F).

Recall In CLIP, image→text uses a softmax over rows or columns?

Rows ::: Rows — each image is normalized across all candidate captions.

Compare-and-connect: the row/column duality here is the batch-level cousin of metric learning; the augmentation-driven positives of SimCLR come from data augmentation; and the encoders behind $I$ and $T$ are ViT and Transformers.