Self-supervised and contrastive learning (SimCLR, CLIP): Visual walkthrough

This page is the visual companion to Self-supervised and contrastive learning (SimCLR, CLIP). If you have seen Metric learning and embedings the idea of "distance between points that mean things" will feel familiar; if not, we build it here from scratch.

Step 1 — A representation is just an arrow to a point

WHAT. A neural network takes an image and spits out a list of numbers, like $z = (0.7, 0.2)$ . We are going to use only 2 numbers so we can draw it. A list of 2 numbers is a point on a flat sheet of paper — or equivalently an arrow from the centre (called the origin) out to that point.

WHY. Everything in contrastive learning is about where points land. Before we can say "pull these together, push those apart", we need a place where points live. That place is called embedding space, and here it is just the 2D page you are looking at.

PICTURE. Look at the figure. Three images (a cat photo, a second crop of the same cat, and a dog) each become one arrow. The symbol $z$ means "the arrow for one image". A little superscript will soon tell us which kind of image it is.

Figure — Self-supervised and contrastive learning (SimCLR, CLIP)

Step 2 — We only care about direction, so we measure angle

WHAT. Take two arrows $u$ and $v$ . We ask: do they point the same way? The single number that answers this is the cosine of the angle $θ$ between them: $sim (u, v) = \frac{u ^{⊤} v}{∥ u ∥ ∥ v ∥} = cos θ$

Term by term:

$u^{⊤} v$ (say "u dot v") — multiply matching numbers and add: $u_{1} v_{1} + u_{2} v_{2}$ . This is the dot product. It is big when the arrows lean the same way.
$∥ u ∥$ (say "length of u") — how long the arrow is, $u_{1}^{2} + u_{2}^{2}$ . Dividing by the two lengths cancels length out, leaving only direction.
$cos θ$ — the cosine of the angle. It runs from $+ 1$ (same direction, $θ = 0°$ ) through $0$ (perpendicular, $θ = 90°$ ) down to $- 1$ (opposite, $θ = 180°$ ).

WHY this tool and not plain distance? We could measure straight-line distance between the two points. But a cat photo and its brighter crop might land at different distances from the origin (one arrow longer than the other) while still meaning the same thing. Direction survives brightness/scale changes; length does not. So we throw length away on purpose and keep the angle. That is why cosine, not Euclidean distance.

PICTURE. Two arrows and the wedge $θ$ between them. As $θ$ shrinks to $0$ , $cos θ$ climbs to $1$ ; as they splay apart, it falls. The green wedge is small for the cat–cat pair, wide for the cat–dog pair.

Formula Cosine similarity, all cases

angle $θ$	arrows	$cos θ$	meaning
$0°$	identical direction	$+ 1$	"definitely same"
$90°$	perpendicular	$0$	"unrelated"
$180°$	opposite	$- 1$	"definitely different"

similarity of two identical-direction vectors

+ 1

similarity of two opposite vectors

- 1

Step 3 — Name the players: anchor, positive, negatives

WHAT. Pick one arrow and call it the anchor, written $z$ . The arrow that should be its friend (a different crop of the same image) is the positive, written $z^{+}$ . Every other arrow in the batch is a negative, written $z_{i}^{-}$ , where $i = 1, 2, \dots, N$ counts them.

WHY. We cannot say "pull together" and "push apart" until we have named what goes where. The plus and minus superscripts are the whole vocabulary: $+$ = "belongs with me", $-$ = "does not".

PICTURE. One blue anchor arrow. One orange positive nearby (small angle — good). Several red negatives fanned out at wide angles. This is the entire cast of the loss function on one page.

Step 4 — Turn "similar or not" into a guessing game (softmax)

WHAT. Line up $N + 1$ similarity scores: one for the positive, $sim (z, z^{+})$ , and one for each negative, $sim (z, z_{i}^{-})$ . Now imagine a quiz: "Which of these $N + 1$ arrows is the true partner of the anchor?" We want the model to point confidently at the positive.

To turn scores into a probability of guessing correctly, we exponentiate each score and divide each by the total. This machine is called softmax: $p (positive is the partner) = \frac{e x p ( sim ( z , z ^{+} ))}{the true one exp ( sim ( z , z ^{+} )) + all the imposters i = 1 \sum N exp ( sim ( z , z _{i}^{-} ))}$

Term by term:

$exp (\cdot)$ — the exponential $e^{(\cdot)}$ . It is always positive and it exaggerates gaps: a score of $0.8$ becomes much bigger than a score of $0.2$ . That is exactly the "amplify the winner" behaviour we want. (We use $exp$ and not, say, squaring, because it turns addition of scores into multiplication of probabilities and never goes negative — perfect for probabilities.)
numerator — the true partner's exponentiated score.
denominator — the true one plus all imposters, so the fraction is a genuine probability between $0$ and $1$ (it can never exceed the whole sum it sits inside).

WHY a probability? Because now we can grade the model with one clean rule: reward it for putting probability on the truth. That converts a fuzzy geometric wish ("point the same way") into a single number to optimise.

PICTURE. A bar chart: the tall orange bar is the positive's $exp$ score; short red bars are negatives. Softmax = "orange bar's height ÷ total height of all bars". You can literally see the probability as the orange slice of the whole.

Step 5 — The temperature knob $τ$ : how picky is the quiz?

WHAT. Before exponentiating, divide every similarity by a small positive number $τ$ (Greek "tau"), the temperature: $\frac{e x p ( sim ( z , z ^{+} ) / τ )}{e x p ( sim ( z , z ^{+} ) / τ ) + \sum _{i} e x p ( sim ( z , z _{i}^{-} ) / τ )}$

Term by term:

$/ τ$ — divides each score. Because $τ$ is small (like $0.1$ ), dividing magnifies every score. A gap of $0.6$ between positive and negative becomes a gap of $6$ after dividing by $0.1$ — huge once exponentiated.

WHY. $τ$ controls how sharp the softmax is:

Small $τ$ ( $0.1$ ): scores get stretched apart → the winner dominates → the model is forced to be very confident and pays extra attention to the hardest (closest) negatives.
Large $τ$ ( $1.0$ ): scores stay close → probabilities flatten → all negatives treated gently.

PICTURE. The same bar chart at three temperatures. At $τ = 1$ the bars are comparable (soft). At $τ = 0.1$ the orange bar towers over a flattened field of red (sharp). One knob, sliding the whole distribution between "relaxed" and "ruthless".

τ

is very small, the model focuses on...

the hardest (most similar) negatives

Step 6 — Grade the guess: $- lo g$

WHAT. We have a probability $p$ of guessing the positive. A perfect model gives $p = 1$ ; a clueless one gives $p$ near $0$ . We turn "how good is this probability?" into a loss to minimise by taking the negative logarithm: $L = - lo g p$

Term by term:

$lo g$ — the logarithm. As $p$ goes from $1$ down toward $0$ , $lo g p$ goes from $0$ down toward $- \infty$ .
the minus sign — flips it so loss is $0$ when $p = 1$ (perfect) and grows huge as $p \to 0$ (terrible). Minimising this loss is maximising the probability.

WHY $- lo g$ and not $1 - p$ ? The log punishes confident mistakes brutally: being sure and wrong ( $p$ tiny) sends the loss toward infinity, producing a big corrective gradient. A plain $1 - p$ tops out at $1$ and barely nudges. Log also turns the product of many examples' probabilities into a clean sum, which is numerically stable and easy to differentiate.

PICTURE. The curve $- lo g p$ from $p = 0$ to $p = 1$ : it kisses zero at the right (perfect) and rockets up on the left (disaster). A dot slides down the curve as training improves.

Assembling Steps 4–6 gives back exactly the parent's formula: $L_{contrastive} = - lo g \frac{exp ( sim ( z , z ^{+} ) / τ )}{exp ( sim ( z , z ^{+} ) / τ ) + \sum _{i = 1}^{N} exp ( sim ( z , z _{i}^{-} ) / τ )}$ Nothing here is arbitrary anymore — every piece answered a question.

Step 7 — Watch one gradient step move the arrows

WHAT. Plug in numbers like the parent's worked example. Anchor–positive similarity $0.85$ , six negatives around $0.2$ , temperature $τ = 0.5$ .

Numerator: $exp (0.85/0.5) = exp (1.7) \approx 5.47$ .
Denominator: $5.47 +$ (six negative terms $\approx exp (0.4) \approx 1.49$ each) $\approx 5.47 + 8.9 \approx 14.4$ .
Probability $\approx 5.47/14.4 \approx 0.38$ , loss $\approx - lo g (0.38) \approx 0.97$ .

WHY show the geometry? Because minimising that $0.97$ does something you can draw: gradient descent (see Batch normalization and training techniques for the training machinery) rotates the anchor toward the positive and away from each negative, proportional to how confidently each was confused.

PICTURE. Before vs after one update. The blue anchor swings toward the orange positive (angle shrinks, $cos$ rises) and the red negatives drift to wider angles. This is "pull together, push apart" made mechanical.

Step 8 — Edge and degenerate cases (never get surprised)

WHAT & WHY. A good understanding covers what happens at the extremes:

Zero negatives ( $N = 0$ ). Denominator = numerator, so $p = 1$ , loss $= 0$ always. Nothing to learn — the model can place all arrows identically and never be corrected. Lesson: negatives are what create the "push apart" force; that is why Metric learning and embedings and SimCLR crave many negatives (large batches).
Positive collapses onto a negative (some $z_{i}^{-}$ points the same way as $z^{+}$ ). Then two big terms fight in the denominator, $p$ drops, loss spikes — the loss complains loudly, exactly the corrective signal we want.
All arrows identical (total collapse). Every similarity equals $1/ τ$ after scaling, positive and negatives indistinguishable, $p = 1/ (N + 1)$ , loss $= lo g (N + 1) > 0$ . The loss stays positive, so gradient descent is pushed to break the tie — good, collapse is penalised.
Temperature $τ \to 0$ . Softmax becomes a hard "winner-takes-all": if any negative outscores the positive even slightly, loss explodes. Too aggressive — training gets unstable. This is why $τ$ lives in $[0.1, 0.5]$ , never at $0$ .

PICTURE. Four mini-panels, one per case: (a) no negatives → flat zero force; (b) a red arrow overlapping the orange → alarm; (c) all arrows stacked → penalised; (d) $τ \to 0$ knife-edge.

The one-picture summary

Everything on this page collapses into a single pipeline: arrows → angles → softmax quiz → $- lo g$ grade → gradient that rotates arrows.

Recall Feynman retelling — say it out loud

Every picture becomes an arrow. I only care which way arrows point, so I measure the angle between them with cosine — same direction gives $+ 1$ , opposite gives $- 1$ . I pick one arrow as the anchor, one true friend (the positive) and a crowd of strangers (negatives). Then I play a quiz: which arrow is the anchor's real friend? Softmax turns the angle-scores into a probability of guessing right, and the temperature $τ$ decides whether the quiz is easygoing or ruthless. I grade the guess with $- lo g$ : perfect probability scores zero loss, a confident wrong guess scores near-infinity. Minimising that loss physically rotates the anchor toward its friend and away from the strangers. Take away the negatives and there is no push, so nothing is learned — which is why big batches matter. Set $τ$ too low and the quiz becomes a hair-trigger and training wobbles. That whole chain — arrow, angle, softmax, $- lo g$ , rotate — is contrastive learning.

Recall Rebuild the formula from scratch

Start from cosine similarity. Divide by $τ$ . Exponentiate. Put the positive on top, the positive-plus-all-negatives on the bottom (a softmax). Take $- lo g$ . Done — that is InfoNCE.

See also: Representation learning · Transfer learning and fine-tuning · Zero-shot and few-shot learning · Multi-modal learning