Foundations — Word embeddings (Word2Vec, GloVe)

This page builds every symbol and idea the parent note leans on, starting from something a 12-year-old can picture, and never using a symbol before it is drawn. Return here whenever a formula upstairs uses notation you don't recognise.

0. A word as a point, before any maths

Forget vocabulary sizes and softmaxes for a moment. Picture a flat sheet of paper. Put a dot on it for the word cat, another dot for dog, another far away for democracy. If cats and dogs behave alike in sentences, their dots should be near each other; democracy should be far off.

Figure 1 — Three words placed as points. The red line joins cat and dog to show "close = similar in meaning"; democracy sits far away. This picture is the whole promise of embeddings: meaning becomes position.

The paper above has 2 directions (left–right, up–down). Real embeddings use 50–300 directions, but we cannot draw 300 axes, so every figure on this page uses 2 as a stand-in. Keep that in mind: the pictures are honest, just squashed.

1. What a vector actually is

A vector is just an ordered list of numbers. "Ordered" means position matters: $[2,5]$ is not the same as $[5,2]$.

Notation you will meet upstairs, decoded:

So when the parent writes ${\mathbf{v}}_{w_t}$, unpack it slowly: $w_t$ is "the word at position $t$ in the sentence", and ${\mathbf{v}}_{w_t}$ is "the arrow we assigned to that word". Nothing more.

2. Dimension — why "50,000" vs "300" matters

The dimension of a vector is simply how many numbers are in the list. A 2-D vector has 2 numbers; a 300-D vector has 300.

The parent note's whole motivation is a fight between two dimensions:

• One-hot vectors live in 50,000-D (see One-hot encoding): a list that is all zeros except a single $1$.
• Embeddings live in ~300-D: a list of ordinary numbers, mostly nonzero.

Figure 2 — Left: a one-hot vector — a tall list that is $0$ everywhere except one red spike. Right: an embedding — a short list of dense, mixed positive/negative numbers. Same word, two representations; the right one is what we want to learn.

Why the topic needs this contrast: in one-hot land, every pair of distinct words is exactly equally far apart — cat–dog and cat–democracy have the same distance. That is the failure embeddings exist to fix.

The symbol $|V|$ means "the size of the vocabulary $V$" — the vertical bars mean "count the things inside". So $|V|=50000$ reads "there are fifty thousand words".

3. The dot product — the machine's ruler for "alike"

Now the single most important operation in the whole topic. Every scoring formula upstairs — ${\mathbf{u}}_w^T\mathbf{h}$, ${\mathbf{u}}_{w_t}^T{\mathbf{v}}_c$, ${\mathbf{w}}_i^T{\tilde{\mathbf{w}}}_j$ — is a dot product. (You met ${\mathbf{u}}_w$, ${\mathbf{v}}_c$ and ${\tilde{\mathbf{w}}}_j$ in Section 1: they are all just word-arrows or context-arrows. Here ${\mathbf{v}}_c$ means "the vector of the specific context word $c$", and $\mathbf{h}$ is defined just below.)

Why this exact tool and not another? We wanted a single number that grows when two arrows point the same way and shrinks (or goes negative) when they point apart. The dot product does precisely that:

• Arrows pointing the same direction → large positive number.
• Arrows at a right angle → exactly $0$.
• Arrows pointing opposite → negative number.

Figure 3 — Three arrows from the origin. The red arrow $\mathbf{h}$ points nearly the same way as the black arrow $\mathbf{u}$, so their dot product is large and positive; the other black arrow leans away, so its dot product with $\mathbf{u}$ is negative. The dot product is the number that reads off this agreement.

That is why "score of word $w$" is written ${\mathbf{u}}_w^T\mathbf{h}$: it measures how much the word's output arrow ${\mathbf{u}}_w$ agrees with the context arrow $\mathbf{h}$. Big agreement → high score → high predicted probability.

4. Cosine similarity — dot product with the size removed

A dot product also grows if you just make the arrows longer, which we don't want — "cat" and "cats" shouldn't rank as more similar merely because someone scaled them. So the topic often measures direction only, using Cosine similarity.

All cases, so you never get surprised:

• $\cos \theta =1$: arrows point the same way (angle $0°$) → maximally similar.
• $\cos \theta =0$: right angle ($90°$) → unrelated.
• $\cos \theta =-1$: opposite ($180°$) → maximally dissimilar.
• One arrow is the zero vector $[0,0,\dots ]$: length is $0$, we'd divide by zero — cosine is undefined. In practice no learned embedding is exactly zero, but this is the degenerate case to remember.

5. Turning scores into probabilities — the softmax

The dot-product scores are raw numbers, possibly negative, that don't add to 1. To pick a word we want probabilities: all non-negative, summing to 1. That is the job of the Softmax function.

Two symbols to earn here:

• $\exp (x)$, also written $e^x$, is the exponential: a function that turns any number into a positive one and grows very fast. We use it because we need every score, even a negative one, to become a positive weight — probabilities can't be negative.
• The denominator ${\sum }_w\exp (s_w)$ adds up everyone's positive weight, so dividing by it forces the outputs to sum to exactly 1.

Figure 4 — Left: raw dot-product scores, some negative. Right: after softmax every bar is positive and the bars sum to 1; the tallest score becomes the red "winner" probability. Softmax keeps the ranking but reshapes scores into a probability distribution.

6. The sigmoid — softmax's two-word cousin

Negative sampling replaces the giant softmax with many tiny yes/no questions. Each yes/no question uses the sigmoid.

Handy fact the parent uses: $\sigma (-x)=1-\sigma (x)$. So "probability a fake pair is fake" is $\sigma (-\text{score})$, which is why negative samples appear with a minus sign upstairs.

7. Probability notation $P(w_t\mid \text{context})$

The bar $\mid$ means "given". $P(w_t\mid \text{context})$ reads "the probability of the word $w_t$, given that we already know the surrounding context". It is a number between 0 and 1.

• $\log P(\cdots )$: the logarithm of that probability. In this topic (and machine learning generally) $\log$ always means the natural logarithm, base $e\approx 2.718$ — the same $e$ that appears in $\exp$; it is the exact inverse of $\exp$, so $\log (\exp (x))=x$. Log turns multiplication into addition (so a product over a whole corpus becomes a friendly sum) and is always negative for probabilities under 1 — that's why the loss upstairs carries a minus sign to make it positive.
• ${\prod }_{t=1}^T$: the product symbol (capital pi) — "multiply all of these together", the multiply-version of $\sum$.

8. Co-occurrence counts — the raw fuel of GloVe

GloVe never looks at arrows first; it starts from counting. The symbol $X_{ij}$ is simply how many times word $j$ appeared near word $i$ across the whole corpus.

Everything GloVe does — ratios of $P_{ij}$, the log, the weighting $f(X_{ij})$ — is arithmetic on this table. You now hold every symbol it needs (including the tilde in ${\tilde{\mathbf{w}}}_j$, decoded back in Section 1).

Prerequisite map

The diagram below is written in mermaid, a text-to-diagram format. Read each arrow A --> B as "idea A feeds into idea B"; if your reader renders mermaid you will see labelled boxes joined by those arrows, all flowing down into the parent topic at the bottom.

This feeds straight into the parent topic, and from there onward into Transfer learning, Attention mechanism, BERT, and visualising results with t-SNE.

Equipment checklist

Each line below is written as question ::: answer. The ::: simply splits the line: everything to its left is a question, everything to its right is the answer. Cover the right side, answer out loud, then reveal. If any line stumps you, re-read its section.