Multimodal models (vision-language): Visual walkthrough

This page rebuilds the central engine of a vision-language model — the contrastive loss that glues pictures to words — one picture at a time. We start with nothing but "here is an image, here is a caption" and end with a working training signal. Every symbol is earned before it is used.

Step 1 — Two data types that cannot be compared yet

WHAT. We start with a single training example: one image and one caption that describe the same thing (say, a photo of a dog and the words "a golden retriever").

WHY. Before we can teach a model that these two things mean the same, we must notice the honest obstacle: an image is a grid of numbers (brightness of each pixel) and text is a list of word-pieces. There is no way to subtract a sentence from a photo. They live in different worlds.

PICTURE. On the left, a picture is a stack of pixel numbers with height $H$ , width $W$ , and $3$ colour channels (red, green, blue). On the right, text is a short chain of tokens. Nothing connects them.

These live in different shapes, so direct comparison is impossible. That single fact is the whole reason for everything that follows.

Step 2 — Turn each world into an arrow (a vector)

WHAT. Feed the image through a vision encoder and the text through a transformer text encoder. Each spits out a list of numbers — a vector, which you should picture as an arrow pointing somewhere.

WHY. Two objects of different shape can never be compared. But two arrows of the same length can — you can ask "do they point the same way?". So the plan is: convert both worlds into arrows that live in one shared space of dimension $d$ (e.g. $d = 512$ numbers). This shared space is the reusable "meaning space".

PICTURE. The image-encoder produces an arrow; the text-encoder produces another arrow; both are dropped into the same $d$ -dimensional space. They may point in totally unrelated directions at first — training has not happened yet.

The two separate encoders exist because images want 2D patch structure and text wants 1D token structure — different inductive biases. Only after encoding do they meet.

Step 3 — Measure "same direction" with cosine similarity

WHAT. Given two arrows, we score how aligned they are with cosine similarity — literally the cosine of the angle between them.

WHY. We do not care how long an arrow is, only which way it points. A dog is a dog whether the encoder shouts it (long arrow) or whispers it (short arrow). The tool that reads direction while ignoring length is the cosine of the angle. That is exactly why we pick cosine and not plain distance.

PICTURE. Two arrows meet at an angle $θ$ . When they overlap ( $θ = 0°$ ) the score is $+ 1$ . At a right angle ( $90°$ ) it is $0$ . Pointing opposite ( $180°$ ) gives $- 1$ .

Degenerate case — the zero arrow. If an arrow has length $0$ (all its numbers are zero), then $∥ a ∥ = 0$ and we would divide by zero. In practice encoders never output the exact zero arrow, but this is why implementations add a tiny $ε$ in the denominator: $∥ a ∥ + ε$ . The cosine of a zero-length arrow is simply undefined — a point has no direction.

Step 4 — One image against a whole batch of captions

WHAT. Take a batch of $B$ pairs. Fix one image (say image $1$ ). Compute its cosine similarity against every caption in the batch: its own caption and everyone else's.

WHY. A single "these match: good" score is not enough to learn from — the model could cheat by making everything match. We need pressure in two directions at once: pull the true caption in, and push the $B - 1$ wrong captions away. Comparing against the whole batch gives us those wrong examples for free — they are called negatives.

PICTURE. A similarity matrix: rows are images, columns are captions. The diagonal cells are the true pairs (image $i$ with its own caption $t_{i}$ ) — we want those bright. The off-diagonal cells are mismatches — we want those dim.

Every row of the matrix is now a little multiple-choice quiz: "which of these $B$ captions is yours?"

Step 5 — Turn the row into probabilities (softmax)

WHAT. Convert image $i$ 's row of similarity scores into a probability distribution — one number per caption, all positive, summing to $1$ — using the softmax function.

WHY. A raw score like $0.9$ is not a probability. But if we phrase training as "pick the correct caption out of $B$ ", that is exactly a classification problem, and classification wants probabilities so we can use cross-entropy loss. The softmax is the standard bridge from "scores" to "probabilities". We exponentiate to force every number positive, then divide by the total so they sum to one.

PICTURE. The row of scores becomes tall/short bars; softmax reshapes them into a set of bars that add up to a full bar of height $1$ . The exponential stretches gaps — a small lead in score becomes a large lead in probability.

Step 6 — Make it symmetric (both directions)

WHAT. Everything above quizzed images picking captions (rows). Do the mirror quiz too: captions picking images (columns). Average the two losses.

WHY. Alignment is a two-way street. If "cat photo" should find "cat text", then "cat text" should equally find "cat photo". Training only one direction lets the model cheat by collapsing all captions onto one point — a degenerate solution. Forcing both directions blocks that escape.

PICTURE. The same matrix, read two ways: softmax along each row (image→text) and softmax down each column (text→image). Both want the diagonal bright.

Step 7 — What one training step actually moves

WHAT. Backpropagation nudges every arrow: it pulls each true pair together and pushes every false pair apart, all in one gradient step.

WHY. This is where the abstract loss becomes geometry. A high loss means "diagonal not bright enough" — so the update rotates the correct image and text arrows toward each other, and rotates the mismatched ones away. Repeated over billions of pairs, this carves out a space where meaning determines direction. Contrastive loss is famously noise-tolerant: even messy web captions work, because as long as true pairs are slightly more similar than false ones, the signal accumulates.

PICTURE. Before: arrows scattered. After one step: the dog-image and "dog" arrows swing closer; the dog-image and "car" arrows swing apart.

The one-picture summary

This single figure compresses all seven steps: two worlds → two encoders → two arrows → a similarity matrix → softmax both ways → a pull-and-push update.

Recall Feynman retelling (say it out loud, no symbols)

A picture and a sentence live in totally different shapes, so you cannot compare them directly. So you run each through its own machine that turns it into an arrow in one shared space. Two arrows can be compared — you just check whether they point the same way, which is what cosine similarity measures (it ignores how long the arrows are, only their direction). Now take a whole batch of picture-sentence pairs and score every picture against every sentence. Each picture should point most like its own sentence and least like the others. Turn each picture's row of scores into probabilities with softmax (exponentiate to make them positive, divide by the total to make them sum to one), then the loss just says "how confident were you in the right sentence?" — confident means low loss. Do the same quiz the other way (sentences picking pictures) and average, so nothing can cheat by collapsing. Every training step then pulls true pairs together and shoves false pairs apart. After enough of this, meaning becomes direction — and a fresh photo automatically lands next to the word that describes it, letting you name things you never trained on.

Recall Quick self-test

Why cosine similarity and not plain distance? ::: Because we care about direction (which way an arrow points = what it means), not magnitude; cosine divides out length so only angle matters. What are the "negatives" in the batch? ::: The $B - 1$ captions that do not belong to a given image; the loss pushes them away. What does small temperature $τ$ do? ::: It sharpens the distribution — small score gaps become huge probability gaps, forcing the model to be confident. Why make the loss symmetric? ::: So the model cannot cheat by collapsing all captions to one point; both image→text and text→image quizzes must succeed. Why does a zero-length arrow break cosine similarity? ::: A point has no direction, and the formula divides by the length, so you would divide by zero — it is undefined.