Word embeddings (Word2Vec, GloVe): Visual walkthrough

This page rebuilds ONE result from nothing: the negative sampling training rule that Word2Vec's Skip-gram actually uses. The parent note gave you the final loss. Here we earn every symbol, one picture at a time, until that loss stops looking like magic and starts looking obvious.

Step 1 — A word is a labelled slot, nothing more

WHAT. Before meaning, a computer only knows identity. If our vocabulary has $∣ V ∣$ distinct words, we give each word one slot. "cat" might be slot 3; "dog" slot 7. We write a word as a column of numbers that is all $0$ except a single $1$ in that word's slot. This is One-hot encoding.

WHY. We need some number-form to start. One-hot is the most honest one: it claims nothing except "this is word number $k$ ". No word is closer to any other — every pair is equally far apart. That emptiness is exactly the problem embeddings will fix.

PICTURE. Look at the tall thin column in the figure: one amber $1$ , a stack of white $0$ s. Every word gets its own column, and geometrically they sit on perpendicular axes — no two share a direction.

Figure — Word embeddings (Word2Vec, GloVe)

Step 2 — Two lookup tables turn a slot into a location

WHAT. We introduce two matrices. Call them $W$ (the input/word table) and $U$ (the output/context table). Each has $∣ V ∣$ rows and $d$ columns, where $d$ is a small number like $100$ . Row $k$ of $W$ is the word's own vector $v_{w} \in R^{d}$ ; row $k$ of $U$ is the word's context vector $u_{w} \in R^{d}$ .

WHY. We want each word to live at a point in $d$ -dimensional space so that nearness can mean "similar meaning". Multiplying a one-hot vector by $W$ just selects one row — so $v_{w} = W^{⊤} e_{w}$ is nothing but a lookup. We keep two vectors per word (one as "I am the centre word", one as "I am a neighbour") because a word playing centre and the same word playing neighbour should be allowed to behave differently; it makes the maths and the gradients cleaner.

PICTURE. The one-hot spike acts like a finger pointing at exactly one row of the grid $W$ ; that row slides out as the dense vector on the right.

Step 3 — The learning signal: real neighbours in the text

WHAT. Slide a small window across the corpus. At each position we have a centre word $w_{t}$ and its context words $w_{t + j}$ (the words within $C$ steps to the left and right, $j \neq = 0$ ). Each pair $(w_{t}, w_{t + j})$ that actually occurs is a positive example.

WHY. Skip-gram's whole bet, from the parent note, is: a word is known by the company it keeps. If we can make each centre word good at predicting its true neighbours, its vector must absorb what those neighbours have in common — its meaning. So the raw material for learning is simply: which words really sat next to which.

PICTURE. The amber box is the centre word "sat"; the cyan boxes inside the window are its true context. Each arrow "sat → the", "sat → cat", "sat → on", "sat → mat" is one positive training pair.

Notice the payoff the parent mentioned: one centre word spawns several pairs (one per neighbour), so even a rare word gets multiple updates.

Step 4 — Scoring a pair with a dot product

WHAT. Given a centre vector $v_{c}$ and a candidate context vector $u_{o}$ , we measure how well they match with a dot product: $s = u_{o}^{⊤} v_{c} = \sum_{m = 1}^{d} u_{o} [m] v_{c} [m] .$

WHY this tool. We need one number saying "do these two vectors point the same way?" The dot product is the natural answer: it is large and positive when two vectors point together, near zero when perpendicular, and negative when opposed. It is the un-normalised cousin of Cosine similarity — same sign behaviour, without dividing by lengths. We use the raw dot product (not full cosine) because it is cheap and the network can absorb the scaling into the vectors themselves.

PICTURE. Two arrows from the origin: when the angle between them is small the projection (the amber drop-line) is long → big score; when they are at right angles the projection collapses to zero.

Step 5 — Squashing a score into a probability with the sigmoid

WHAT. A score $s$ can be any real number (from $- \infty$ to $+ \infty$ ). We want a probability — a number in $[0, 1]$ answering "is this a real pair?". The sigmoid does exactly this: $σ (s) = \frac{1}{1 + e ^{- s}} .$

WHY the sigmoid and not softmax. Softmax (used in the parent's plain Skip-gram) needs the scores of all $∣ V ∣$ words in its denominator — brutally expensive. The trick of negative sampling is to stop asking "which of the 50,000 words is the neighbour?" and instead ask a much cheaper yes/no question about one pair at a time: "real or fake?" A yes/no question needs a squasher that maps one score to one probability — that is the sigmoid. It hands us $σ (s) \to 1$ for large positive $s$ , $σ (s) \to 0$ for large negative $s$ , and $σ (0) = 0.5$ for "no idea".

PICTURE. The S-curve: horizontal axis is the score $s$ , vertical axis is probability. Watch it hug $1$ on the right, $0$ on the left, pass through $0.5$ at the origin.

Step 6 — Faking neighbours: negative sampling

WHAT. For each real pair $(c, o)$ we invent $k$ fake pairs $(c, w_{i})$ by drawing $k$ random words $w_{1}, \dots, w_{k}$ from a noise distribution $P_{n} (w)$ . These are almost certainly not real neighbours of $c$ .

WHY. The sigmoid only knows "real vs fake". If we only ever showed it real pairs, it could cheat by calling everything real (predict $1$ always). We must show it counter-examples so it learns to say "fake" too. Drawing them at random is cheap, and with $k$ around $5$ – $20$ it is enough. The noise distribution is the smoothed unigram $P_{n} (w) \propto f (w)^{0.75},$ where $f (w)$ is word $w$ 's raw frequency. The exponent $0.75$ pulls very common words down and lifts rare words up, so negatives aren't only "the", "of", "a".

PICTURE. Around the true arrow "sat → cat", four amber dashed arrows fly off to random words "banana", "of", "planet", "the" — the sampled negatives we will push away.

Step 7 — Assemble the objective, term by term

WHAT. We want, for each centre word $c$ :

the real neighbour $o$ to score high → $σ (u_{o}^{⊤} v_{c})$ near $1$ ;
each fake neighbour $w_{i}$ to score low → its fake probability $σ (- u_{w_{i}}^{⊤} v_{c})$ near $1$ .

Maximising both means maximising their product. Taking $- lo g$ (turns a product into a sum and turns "maximise" into "minimise a loss") gives the objective the parent stated:

WHY this shape. $- lo g (prob)$ is $0$ when the probability is $1$ and blows up toward $+ \infty$ as it drops to $0$ . So the loss is small exactly when we got both jobs right, and painful when we didn't. That is precisely what an optimiser wants to push downhill.

PICTURE. Two stacked bars: the real term shrinks its loss by raising $σ$ toward the top; each fake term shrinks its loss by raising $σ (- s)$ — i.e. driving its score down. Green = "pull together", amber = "push apart".

Step 8 — What one gradient step actually does to the arrows

WHAT. Backpropagation on this loss gives beautifully simple updates. Using $σ^{'} (s) = σ (s) (1 - σ (s))$ , the gradient w.r.t. the centre vector is $\frac{\partial L}{\partial v _{c}} = (σ (u_{o}^{⊤} v_{c}) - 1) u_{o} + \sum_{i = 1}^{k} σ (u_{w_{i}}^{⊤} v_{c}) u_{w_{i}} .$

WHY it's intuitive. For the real pair, $(σ (s) - 1)$ is negative (since $σ < 1$ ), and gradient descent steps opposite the gradient — so we move $v_{c}$ toward $u_{o}$ : pull real neighbours together. For each fake pair, $σ (s)$ is positive, so we move $v_{c}$ away from $u_{w_{i}}$ : push fakes apart. The size of each nudge is the error $∣ σ (s) - target ∣$ : pairs already correct barely move, wrong ones move a lot.

PICTURE. The centre arrow "sat" slides a short green step toward "cat" and short amber steps away from the four random negatives — the physical meaning of one update.

The one-picture summary

Everything above compresses to a single tug-of-war around the centre word: green ropes pull the true neighbour in, amber ropes shove the random negatives out, and the sigmoid decides how hard to pull. Repeat over the whole corpus and the geometry that survives is meaning.

Recall Feynman retelling — say it back in plain words

We started with words as dumb labels — one $1$ in a long stack of $0$ s, every word equally far from every other. We handed each word two secret coordinates lists (a "me" vector and a "neighbour" vector). To score whether two words belong together, we point their arrows and take a dot product — long shadow means good match. We squashed that score through the S-curve so it reads as a probability: "is this a real neighbour, yes or no?" To stop the model from lazily shouting "yes" to everything, we invented fake neighbours by picking random words (leaning slightly toward rarer ones via the $0.75$ power). Then the loss is just: make real pairs read near 1, make fake pairs read near 0, and $- lo g$ punishes us whenever we don't. One gradient step literally slides the centre word's arrow toward its true neighbour and away from the fakes, by an amount equal to how wrong we were. Do this millions of times and words that keep the same company drift into the same neighbourhood — that neighbourhood is the embedding space.

Recall Quick self-test

Why sigmoid instead of softmax here? ::: Softmax's denominator sums over all $∣ V ∣$ words (expensive); negative sampling replaces the "which word?" question with cheap yes/no "real or fake?" questions, each answered by one sigmoid. What does $σ (- s)$ represent? ::: The probability the pair is fake, using the identity $1 - σ (s) = σ (- s)$ . Which direction does $v_{c}$ move for a real pair? ::: Toward the true context vector $u_{o}$ , because the factor $σ (s) - 1$ is negative and descent steps opposite the gradient. Why exponent $0.75$ in the noise distribution? ::: It down-weights very frequent words and up-weights rare ones, so negatives are informative rather than all being "the/of/a". What breaks if $k = 0$ ? ::: Nothing stops all vectors collapsing together — negatives are what prevent trivial collapse.

Related paths onward: Cosine similarity (how we read out the learned space), t-SNE (how we see it in 2-D), Transfer learning and Attention mechanism / BERT (what these static vectors grow into).