Worked examples — Word embeddings (Word2Vec, GloVe)

This page takes the machinery from Word embeddings (Word2Vec, GloVe) and runs the numbers — by hand, on tiny vocabularies you can hold in your head. The parent gave you the formulas. Here we plug in real numbers and check every corner case: the friendly ones, the degenerate ones (zeros, ties), and the exam-trap ones.

Before we start, one promise: every symbol below is re-explained the first time it appears, and every number is verified at the bottom.

---

What the symbols mean (built from zero)

We will need three helper functions repeatedly:

Two more formulas, stated now so no example uses them cold:

---

The scenario matrix

Every problem this topic can throw at you falls into one of these cells. The examples below are tagged with the cell(s) they cover.

Cell	Case class	The tricky bit
A	CBOW forward pass, ordinary inputs	averaging context, softmax normalises
B	Skip-gram forward pass	one target → many context probs, no averaging
C	Negative-sampling loss, real pair	$\sigma$ near 1 → loss near 0
D	Negative-sampling loss, fake pair	uses $\sigma (-x)$; sign flip matters
E	Degenerate: zero score (${\mathbf{u}}^T\mathbf{v}=0$)	$\sigma (0)=0.5$, uniform softmax
F	Degenerate: tie / identical arrows	softmax gives equal probs; cosine $=1$
G	GloVe target for a pair	$\log X_{ij}$, and the $X_{ij}=0$ trap
H	GloVe weighting $f(x)$ across all 3 regimes	$x<x_{\max }$, $x=0$, $x\ge x_{\max }$
I	Limiting behaviour	huge/tiny scores → softmax saturates
J	Word-analogy word problem	king − man + woman, cosine ranking
K	Exam twist	negative-sampling gradient sign

Nine worked examples below hit all eleven cells.

---

Example 1 — CBOW forward pass (Cell A)

Forecast: Guess — will "sat" get more or less than a uniform $\frac{1}{4}=0.25$? Jot it down.

1. Average the context arrows to get the hidden vector $\mathbf{h}$. $\mathbf{h}=\frac{1}{2}({\mathbf{v}}_{\text{cat}}+{\mathbf{v}}_{\text{on}})=\frac{1}{2}((1,0)+(0,1))=(0.5,0.5)$ Why this step? CBOW's hidden layer is literally the average of the context embeddings — it blends the surrounding words into one "context mood" arrow. See the figure below — the two chalk-blue context arrows and their yellow average.

Figure 1 — CBOW hidden vector $\mathbf{h}$ (yellow) is the average of the two context arrows (blue). It lands exactly midway between ${\mathbf{v}}_{\text{cat}}$ and ${\mathbf{v}}_{\text{on}}$.

2. Score every vocabulary word with $\text{score}(w)={\mathbf{u}}_w^T\mathbf{h}$. \text{cat}&: (1,1)^T(0.5,0.5)=0.5+0.5=1.0\\ \text{sat}&: (2,0)^T(0.5,0.5)=1.0+0=1.0\\ \text{on}&: (0,2)^T(0.5,0.5)=0+1.0=1.0\\ \text{mat}&: (1,-1)^T(0.5,0.5)=0.5-0.5=0.0 \end{aligned}$$ *Why this step?* The dot product asks "how aligned is each output arrow with the context mood?" Higher = more likely to be the missing word.
3. Softmax the scores $s=(1,1,1,0)$. $e^1=2.71828,e^0=1,\text{denominator}=3e^1+1=9.15484$ $P(\text{sat})=\frac{e^1}{3e^1+1}=\frac{2.71828}{9.15484}=0.29692$

Verify: All four probabilities must sum to 1: $3(0.29692)+0.10924=1.0$. ✓ And "sat" beats uniform $0.25$ because its score (1.0) is above the low scorer "mat" (0.0). Did your forecast agree?

---

Example 2 — Skip-gram forward pass (Cell B)

Forecast: Skip-gram uses no averaging — the hidden vector is just ${\mathbf{v}}_{\text{sat}}$. Which context word wins, cat or mat?

1. Hidden vector = the target arrow, untouched: $\mathbf{h}={\mathbf{v}}_{\text{sat}}=(1,1)$. Why? Skip-gram feeds one word in, so there is nothing to average — this is the whole architectural difference from CBOW.

2. Score all words $\text{score}(w)={\mathbf{u}}_w^T(1,1)$: $\text{cat}=(1,1{)}^T(1,1)=2,\text{sat}=(2,0{)}^T(1,1)=2,\text{on}=(0,2{)}^T(1,1)=2,\text{mat}=(1,-1{)}^T(1,1)=0$ Why this step? Just like CBOW, the dot product measures how aligned each output arrow ${\mathbf{u}}_w$ is with the target's hidden vector — "how likely is $w$ to appear near sat?" Higher score = more likely a context word.

3. Softmax, scores $(2,2,2,0)$: denominator $=3e^2+1=3(7.38906)+1=23.16717$. $P(\text{cat}\mid \text{sat})=\frac{e^2}{23.16717}=\frac{7.38906}{23.16717}=0.31894$ $P(\text{mat}\mid \text{sat})=\frac{1}{23.16717}=0.04316$ Why softmax? We need the scores turned into a probability distribution over the whole vocabulary (they must sum to 1) so we can read off "probability that this word is a context word." Softmax is the tool that normalises raw scores into probabilities.

Verify: cat (0.319) $\gg$ mat (0.043), and cat/sat/on tie at 0.319 each. Sum check: $3(0.31894)+0.04316=1.0$. ✓ Cat wins, as forecast.

---

Example 3 — Negative sampling, the real pair (Cell C)

Forecast: A big positive score means the model is confident this real pair is real. Will the loss be near 0 or near $\infty$?

1. Sigmoid of the score: $\sigma (3.0)=\frac{1}{1+e^{-3}}=\frac{1}{1+0.049787}=0.95257$. Why $\sigma$? Negative sampling reframes training as "is this pair real? — a yes/no question." $\sigma$ turns the raw score into $P(\text{real})$.

Figure 2 — the sigmoid curve. The blue dot marks the real pair ($z=3$, $P=0.953$); the pink dot marks a fake pair scored through $\sigma (-z)$ (Example 4); the yellow crosshair marks $\sigma (0)=0.5$ (Example 5).

2. Loss: $-\log (0.95257)=0.04859$.

Verify: $P(\text{real})=0.953$ is close to 1, so the penalty $0.049$ is tiny — exactly what "the model got it right" should look like. ✓

---

Example 4 — Negative sampling, a fake pair (Cell D)

Forecast: The score is positive (+2) — but this pair is supposed to be fake. Note the minus sign inside $\sigma$. Will the loss be big (model is fooled) or small?

1. Flip the sign, then sigmoid: $\sigma (-2.0)=\frac{1}{1+e^2}=\frac{1}{1+7.38906}=0.11920$. Why the minus? $\sigma (-x)=1-\sigma (x)=P(\text{fake})$. We want fake pairs judged fake, so we score $-x$.

2. Loss: $-\log (0.11920)=2.12693$.

Verify: Because the model gave this fake pair a high alignment (+2), it looks too real, so $P(\text{fake})=0.119$ is low and the loss $2.13$ is large — a strong correcting push. That is the desired behaviour: the gradient will pull these arrows apart (this is the Backpropagation signal). ✓ Contrast with Example 3's tiny loss.

---

Example 5 — Degenerate: zero score (Cell E)

Forecast: A zero score means "no opinion — perpendicular arrows, no alignment." What probability does no opinion give?

1. Sigmoid at 0: $\sigma (0)=\frac{1}{1+e^0}=\frac{1}{2}=0.5$. Why it matters: a brand-new, untrained embedding often has near-zero dot products, so the model starts at "50/50 — I don't know." Training moves it off 0.5.

2. Uniform softmax: if all four scores are 0, each probability is $\frac{e^0}{4e^0}=\frac{1}{4}=0.25$. Why? Equal scores → equal probabilities. This is the maximum-uncertainty state.

Verify: $\sigma (0)=0.5$ exactly, and $4\times 0.25=1.0$. ✓ Perpendicular arrows carry zero preference — the geometric meaning of a zero dot product.

---

Example 6 — Degenerate: identical arrows / tie (Cell F)

Forecast: Identical arrows are the "perfect synonym" limit. Guess the cosine and the split.

1. Cosine similarity = $\frac{{\mathbf{a}}^T\mathbf{b}}{\Vert \mathbf{a}\Vert \Vert \mathbf{b}\Vert }$. Here $\mathbf{a}=\mathbf{b}$, so numerator $=\Vert \mathbf{a}{\Vert }^2$ and denominator $=\Vert \mathbf{a}{\Vert }^2$. $\cos =\frac{0.6^2+0.8^2}{\sqrt{0.6^2+0.8^2}\cdot \sqrt{0.6^2+0.8^2}}=\frac{1.0}{1.0}=1.0$ Why cosine, not dot product, for similarity? Cosine ignores arrow length and compares direction only — the right tool when we care about "same meaning" regardless of how frequent (hence how long) a word's vector is.

2. Softmax tie: equal scores over two words → $\frac{e^s}{2e^s}=0.5$ each.

Verify: cosine $=1.0$ (maximum possible — arrows point exactly the same way) and $2\times 0.5=1.0$. ✓ Perfect synonyms sit on top of each other; t-SNE would draw them as one dot.

---

Example 7 — GloVe target, weighting, and the log(0) trap (Cells G, H)

Forecast: Which pairs does GloVe actually train on, and which one gets the flat cap of 1? Guess before computing.

1. Target for ice–solid: the dot-plus-biases should reach $\log X_{ij}=\log 50=3.91202$. Why log? From the parent's derivation, embedding dot products should equal $\log$ of the co-occurrence count — log because meaning-ratios became a homomorphism $F=\exp$, whose inverse is $\log$.

2. Weight $f(50)$, regime $x<x_{\max }$: $f(x)=(x/x_{\max }{)}^{\alpha }=(50/100{)}^{0.75}=0.5^{0.75}=0.59460$. Why weight at all? Frequent pairs would otherwise dominate the loss; $f$ trims rare pairs while still trusting the solid counts.

3. Weight $f(140)$, regime $x\ge x_{\max }$: since $140\ge 100$, $f(140)=1$ exactly — the cap. Why cap? Without it, a hyper-frequent pair like ice–water would drown out everything; the flat 1 stops any single pair from dominating.

4. ice–fashion: $X=0$, so $\log 0=-\infty$ — undefined, would blow up the loss. But $f(0)=(0/100{)}^{0.75}=0$. The weight multiplies the whole squared-error term by zero, so this never-seen pair contributes nothing.

Verify: $\log 50=3.912$, $0.5^{0.75}=0.5946$, $f(140)=1$ (cap regime), and $f(0)=0$ — the zero weight is precisely what defuses the $\log (0)$ trap. ✓

---

Example 8 — Limiting behaviour: softmax saturation (Cell I)

Forecast: A huge lead in score — does softmax give "very likely" or "certainly"? Where does it stop?

1. At $s=(10,0)$: $P_1=\frac{e^{10}}{e^{10}+e^0}=\frac{22026.466}{22026.466+1}=0.99995$. Why this step? To see how fast softmax saturates — a lead of 10 already means 99.995%.

2. At $s=(100,0)$: $e^{100}$ dwarfs $1$, so $P_1\to 1$ but never exactly reaches it (there is always the $+1$). The limit is ${\lim }_{z\to \infty }\frac{e^z}{e^z+1}=1$. Why it matters: huge scores cause numerical overflow in code and vanishing gradients (Backpropagation gets ~0 signal), which is why real implementations subtract the max score before exponentiating.

Verify: $P_1(10,0)=0.99995$ and the limit is exactly $1$ (approached, never hit). ✓

---

Example 9 — Word-analogy word problem + the exam twist (Cells J, K)

Forecast: Everyone "knows" the answer is queen — but prove it with cosine, and beat apple.

1. Vector arithmetic: subtract man, add woman. $\mathbf{t}=\text{king}-\text{man}+\text{woman}=(4,3)-(3,1)+(1,1)=(4-3+1,3-1+1)=(2,3).$ Why subtract man, add woman? Subtracting removes the "male" direction; adding restores a "female" one, keeping the shared "royalty" part. This is the compositionality the parent promised.

Figure 3 — the analogy target $\mathbf{t}$ (dashed pink) equals king − man + woman = (2,3). It lands exactly on queen (yellow) and points nearly opposite to apple.

2. Cosine to queen $=(2,3)$. Since $\mathbf{t}=(2,3)$ is identical to queen, by the Example 6 logic $\cos =1.0$ (numerator $=\Vert \mathbf{t}{\Vert }^2$, denominator $=\Vert \mathbf{t}{\Vert }^2$).

3. Cosine to apple $=(-4,-2)$: ${\mathbf{t}}^T\text{apple}=2(-4)+3(-2)=-8-6=-14,\Vert \mathbf{t}\Vert =\sqrt{2^2+3^2}=\sqrt{13}=3.60555,$ $\Vert \text{apple}\Vert =\sqrt{(-4{)}^2+(-2{)}^2}=\sqrt{20}=4.47214,\cos =\frac{-14}{3.60555\times 4.47214}=\frac{-14}{16.12452}=-\mathrm{0.86824.}$ Why cosine, again? We rank by direction to the target; queen ($+1$) crushes apple ($-0.87$, pointing nearly opposite), so the analogy returns queen.

4. (K-twist) gradient sign. The real-pair loss is $\mathcal{L}=-\log \sigma (z)$ with score $z=-1$. Differentiate: using ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$, $\frac{d\mathcal{L}}{dz}=-\frac{{\sigma }^{\prime }(z)}{\sigma (z)}=-(1-\sigma (z))=\sigma (z)-1.$ Plug $z=-1$: $\sigma (-1)=\frac{1}{1+e^1}=0.26894$, so $\frac{d\mathcal{L}}{dz}=0.26894-1=-0.73106$. What the sign means: a negative gradient of the loss means gradient descent increases $z$ — it pushes the real pair's score up, pulling king/queen-style arrows together. Exactly right for a real pair.

Verify: $\mathbf{t}=(2,3)$; $\cos (\text{queen})=1.0$; $\cos (\text{apple})=-0.86824$; queen wins. Gradient $=\sigma (-1)-1=-0.73106<0$ ⇒ score rises. ✓

---

Recall

Recall Why does CBOW average but Skip-gram not?

CBOW blends many context words into one hidden vector, so it averages. Skip-gram feeds one target word, so there is nothing to average — its hidden vector is the raw input embedding.

Recall What defuses GloVe's

$\log (0)$ problem? The weighting function $f(0)=0$ multiplies the never-co-occurring pair's error by zero, so it never enters the loss.

Recall Real pair vs fake pair in negative sampling — what's the sign trick?

Real pair uses $-\log \sigma (+\text{score})$; fake pair uses $-\log \sigma (-\text{score})$. The minus sign converts $P(\text{real})$ into $P(\text{fake})$.