Word embeddings (Word2Vec, GloVe)

Overview

Word embeddings are dense, low-dimensional vector representations of words that capture semantic meaning and relationships in continuous space. Unlike one-hot encodings (sparse, high-dimensional, no semantic info), embeddings place similar words close together in vector space.

Why Word Embeddings?

The problem with one-hot encoding:

• Vocabulary of 50,000 words → 50,000-dimensional vectors
• Every word equally distant from every other word (no semantic similarity)
• "cat" and "dog" are as different as "cat" and "democracy

What embedings fix:

1. Dense representation: 50-300 dimensions instead of 50,000
2. Semantic similarity: Similar words have similar vectors (measured by cosine similarity)
3. Compositionality: Vector operations capture meaning relationships
4. Transferability: Pre-train on large corpus, use in downstream tasks

Word2Vec: Learning from Context

Word2Vec has two architectures:

1. CBOW (Continuous Bag of Words)

CBOW predicts a target word from its surrounding context words.

How it works:

1. Input Layer: One-hot vectors for context words

2. Hidden Layer: Average the context word embedings $\mathbf{h} = \frac{1}{C} \sum_{i=-C, i \neq 0}^{C} \mathbf{v}_{w_{t+i}}$ where $C$ is context window size, $\mathbf{v}_w$ is embedding for word $w$

3. Output Layer: Compute scores for all vocabulary words $\text{score}(w) = \mathbf{u}_w^T \mathbf{h}$ where $\mathbf{u}_w$ is output embedding for word $w$

4. Softmax: Convert scores to probabilities $P(w_t | \text{context}) = \frac{\exp(\mathbf{u}_{w_t}^T \mathbf{h})}{\sum_{w \in V} \exp(\mathbf{u}_w^T \mathbf{h})}$

Why this works: Words that appear in similar contexts get pushed to similar embedding vectors.

Derivation from first principles:

• Start with goal: maximize probability of observing the corpus
• Corpus probability: $P(\text{corpus}) = \prod_{t=1}^{T} P(w_t | \text{context}_t)$
• Take log (turn product into sum): $\log P(\text{corpus}) = \sum_{t=1}^{T} \log P(w_t | \text{context}_t)$
• Minimize negative log-likelihood (equivalent to maximizing likelihood)
• Normalize by corpus length $T$ for stability

2. Skip-gram

Skip-gram does the reverse: predicts context words given a target word.

How it works:

1. Input: One-hot vector for target word $w_t$
2. Hidden Layer: Just the target word embedding $\mathbf{v}_{w_t}$ (no averaging)
3. Output: Predict each context word independently $P(w_{t+j} | w_t) = \frac{\exp(\mathbf{u}_{w_{t+j}}^T \mathbf{v}_{w_t})}{\sum_{w \in V} \exp(\mathbf{u}_w^T \mathbf{v}_{w_t})}$

Why Skip-gram often works better: Each target word generates multiple training examples (one per context position), so rare words get more updates.

Computational Challenge: Softmax is Expensive

The denominator $\sum_{w \in V} \exp(\mathbf{u}_w^T \mathbf{h})$ requires computing scores for all vocabulary words (often 50,000+) at every training step.

Solution 1: Hierarchical Softmax

Replace flat softmax with a binary tree (Huffman tree):

• Each word is a leaf
• To predict a word, traverse the tree (log₂|V| binary decisions instead of |V| comparisons)
• Probability of a word = product of probabilities along its path

Complexity: $O(\log |V|)$ instead of $O(|V|)$

Solution 2: Negative Sampling

Negative sampling turns the problem into binary classification: "Is this word-context pair real or fake?"

The trick:

• For each real (target, context) pair, sample $k$ random "negative" words (5-20 typically)
• Train to distinguish real pair from negative pairs using logistic regression

Derivation:

• $\sigma(\mathbf{u}_{w_t}^T \mathbf{v}_c)$ = probability real pair is real (should be high)
• $\sigma(-\mathbf{u}_{w_i}^T \mathbf{v}_c) = 1 - \sigma(\mathbf{u}_{w_i}^T \mathbf{v}_c)$ = probability fake pair is fake (should be high)
• Negative samples drawn from noise distribution $P_n(w) \propto f(w)^{0.75}$ (smoothed unigram)
  • Why $0.75$? Balances frequent vs. rare words—without smoothing, would only sample common words

Complexity: $O(k)$ where $k \ll |V|$

GloVe: Global Vectors for Word Representation

Word2Vec is local (looks at small context windows). GloVe is global (uses corpus-wide co-occurrence statistics).

Consider words "ice" and "steam", and context words "solid" vs. "gas":

• $P(\text{solid}|\text{ice})$ is high, $P(\text{solid}|\text{steam})$ is low → ratio is large
• $P(\text{gas}|\text{ice})$ is low, $P(\text{gas}|\text{steam})$ is high → ratio is small
• For unrelated words like "fashion", both probabilities are low → ratio ≈ 1

The idea: Word vectors should encode these ratios.

GloVe Model

Define:

• $X_i = \sum_k X_{ik}$ = total count of all words in context of word $i$
• $P_{ij} = P(j|i) = X_{ij} / X_i$ = probability of word $j$ appearing in context of $i$

Goal: Find embedding vectors such that their dot product relates to log co-occurrence.

Components explained:

• $\mathbf{w}_i$: word embedding for word $i$
• $\tilde{\mathbf{w}_j}$: context embedding for word $j$
• $b_i, \tilde{b}_j$: bias terms for each word
• $f(X_{ij})$: weighting function to handle zero and rare co-occurrences

Why this weighting?

• $f(0) = 0$: Don't train on word pairs that never co-occur (would contribute infinite loss with log(0))
• $f(x) < 1$ for rare pairs: Rare co-occurrences are noisy, downweight them
• $f(x) = 1$ for frequent pairs: Cap the weight so very frequent pairs don't dominate

Derivation from ratio insight:

Start with goal: $\mathbf{w}_i^T \mathbf{w}_j$ should relate to $P_{ij}/P_{ik}$ ratios.

1. Ratio form: $F(\mathbf{w}_i, \mathbf{w}_j, \tilde{\mathbf{w}}_k) = P_{ij}/P_{ik}$ where $F$ is some function
2. Since we want vectors, use dot products: $F(\mathbf{w}_i^T \mathbf{w}_j - \mathbf{w}_i^T \mathbf{w}_k) = P_{ij}/P_{ik}$
3. Homorphism: $F(a - b) = F(a)/F(b)$ suggests $F = \exp$
4. So: $\exp(\mathbf{w}_i^T \mathbf{w}_j) = P_{ij} X_{ij}/X_i$
5. Take log: $\mathbf{w}_i^T \mathbf{w}_j = \log X_{ij} - \log X_i$
6. Absorb $\log X_i$ into bias: $\mathbf{w}_i^T \mathbf{w}_j + b_i + b_j = \log X_{ij}$
7. Distinguish word vs. context embedings (breaks symmetry for training): $\mathbf{w}_i^T \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j = \log X_{ij}$
8. Minimize squared error with weighting: the loss function above

Example 1: CBOW Forward Pass

Sentence: "The cat sat on the mat" Task: Predict "sat" from context ["cat", "on"] (window size = 1) Vocabulary: {the, cat, sat, on, mat} (size5) Embedding dimension: 3

Step 1: Get context embeddings

Embedding matrix E (5 × 3):
"cat" → [0.2, 0.5, 0.1]
"on"  → [0.3, 0.4, 0.2]

Step 2: Average context embeddings $\mathbf{h} = \frac{1}{2}([0.2, 0.5, 0.1] + [0.3, 0.4, 0.2]) = [0.25, 0.45, 0.15]$ Why average? Represents the "central meaning" of the context, order-invariant.

Step 3: Compute output scores

Output matrix W_out (3 × 5):
Scores = W_out^T · h
"the": [0.1, 0.2, 0.3]^T · [0.25, 0.45, 0.15] = 0.025 + 0.09 + 0.045 = 0.16
"cat": [...]^T · h = 0.21
"sat": [...]^T · h = 0.42  ← highest score
"on":  [...]^T · h = 0.18
"mat": [...]^T · h = 0.14

Why dot product? Measures alignment between context representation and candidate word.

Step 4: Softmax $P(\text{sat} | \text{context}) = \frac{\exp(0.42)}{\exp(0.16) + \exp(0.21) + \exp(0.42) + \exp(0.18 + \exp(0.14)} = \frac{1.52}{1.17 + 1.23 + 1.52 + 1.20 + 1.15} = \frac{1.52}{6.27} \approx 0.24$

Why softmax? Converts scores to valid probability distribution (sums to 1).

Example 2: Negative Sampling

Positive pair: (target="cat", context="sat") Task: Train embedings using negative sampling with $k=2$ negatives

Step 1: Compute positive score

v_cat = [0.2, 0.5, 0.1]  (target embedding)
u_sat = [0.3, 0.4, 0.2]  (context embedding)
score_pos = v_cat · u_sat = 0.06+ 0.20 + 0.02 = 0.28

Step 2: Sample negative words Sample 2 random words NOT in context: "the", "mat"

u_the = [0.1, 0.2, 0.3]
u_mat = [0.4, 0.1, 0.2]
score_neg1 = v_cat · u_the = 0.02 + 0.10 + 0.03 = 0.15
score_neg2 = v_cat · u_mat = 0.08 + 0.05 + 0.02 = 0.15

Why random sampling? Efficient approximation of the full vocabulary.

Step 3: Compute loss $\mathcal{L} = -\log \sigma(0.28) - \log \sigma(-0.15) - \log \sigma(-0.15)$ $= -\log(0.57) - \log(0.46 = 0.56 + 0.78 + 0.78 = 2.12$ Interpretation: Positive pair should have high score (close to 1 after sigmoid), negatives should have low score.

Step 4: Gradient update

∂L/∂v_cat pushes v_cat closer to u_sat (positive context)
∂L/∂v_cat pushes v_cat away from u_the, u_mat (negative contexts)

Result: "cat" embedding moves toward words that actually appear near it, away from unrelated words.

Example 3: GloVe Training

Mini co-occurrence matrix:

        cat   sat   mat
dog     5     2     3
cat     08     4

$X_{\text{dog,cat}} = 5$: "cat" appears in context of "dog" 5 times in corpus.

Training pair: (dog, cat) with $X_{ij} = 5$

Current embedings:

w_dog = [0.5, 0.3]
w̃_cat = [0.4, 0.6]
b_dog = 0.1
b̃_cat = 0.2

Step 1: Compute prediction $\hat{X}_{ij} = \exp(\mathbf{w}_{\text{dog}}^T \tilde{\mathbf{w}}_{\text{cat}} + b_{\text{dog}} + \tilde{b}_{\text{cat}})$ $= \exp(0.5 \cdot 0.4 + 0.3 \cdot 0.6 + 0.1 + 0.2) = \exp(0.68) = 1.97$ Why exponent? We're modeling log co-occurrence, so exp converts back to count scale.

Step 2: Compute loss for this pair $f(5) = (5/100)^{0.75} = 0.084$ $\text{loss} = 0.084 \cdot (\log 1.97 - \log 5)^2 = 0.084 \cdot (0.68 - 1.61)^2 = 0.084 \cdot 0.86 = 0.072$ Why squared error? Simple, differentiable objective that drives predicted log-count toward actual log-count.

Step 3: Update embeddings Gradient pushes $\mathbf{w}_{\text{dog}}$ and $\tilde{\mathbf{w}}_{\text{cat}}$ to make their dot product closer to $\log(5) - b_{\text{dog}} - \tilde{b}_{\text{cat}}$.

After many updates: Words that co-occur frequently will have high dot products, rarely co-occurring words will have low dot products.

Properties of Trained Embeddings

Semantic Similarity

Cosine similarity between embedings captures meaning: $\text{similarity}(\mathbf{v}_1, \mathbf{v}_2) = \frac{\mathbf{v}_1 \cdot \mathbf{v}_2}{|\mathbf{v}_1|| \cdot ||\mathbf{v}_2|}$

Similar words have similarity close to 1:

• cos(king, queen) ≈ 0.8
• cos(cat, dog) ≈ 0.76

Analogies

Vector arithmetic captures relationships: $\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}} \approx \mathbf{v}_{\text{queen}}$

Why this works:

• $\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}}$ = "royalty" direction (removes "maleness")
• Adding $\mathbf{v}_{\text{woman}}$ projects onto female royalty

Other examples:

• Paris - France + Italy≈ Rome (capital of...)
• walking - walked + saw ≈ seeing (verb tense)

Clustering

Words cluster by semantic category in embedding space:

• Animals cluster together
• Countries cluster together
• Verbs cluster by aspect/tense

Word2Vec vs. GloVe

In practice: Both produce high-quality embeddings. Word2Vec (especially Skip-gram with negative sampling) is more popular due to speed. GloVe is preferred when you want explicit control over global statistics.

The mistake: "Since 'bank' (financial) and 'bank' (river) are the same word, they'll have the same embedding. Perfect!"

Why it feels right: One word = one vector seems natural.

The problem: Word2Vec and GloVe learn one vector per word type, averaging over all contexts where that word appears. Polysemous words (multiple meanings) get mudy embedings.

The fix:

• Use contextualized embeddings (ELMo, BERT) where "bank" gets different vectors based on context
• Or use sense2vec which clusters word contexts into separatenses

The mistake: Training Word2Vec on 10,000 sentences and expecting good embedings.

Why it feels right: "Deep learning works on small data if I tune carefully."

The problem: Word embedings need millions of word occurrences to capture statistical patterns. With small data:

• Rare words have unreliable embedings (few training examples)
• Semantic relationships are noisy
• Analogies don't work well

The fix:

• Use pre-trained embeddings (trained on billions of words: Google News, Wikipedia, Common Crawl)
• Fine-tune pre-trained embeddings on your domain if needed
• Or use subword embedings (FastText) which handle rare words better

The mistake: Computing $\mathbf{v}_1 \cdot \mathbf{v}_2$ to measure similarity.

Why it feels right: "Bigger dot product = more similar."

The problem: Dot product is affected by vector magnitude, not just direction. A word with high-magnitude embedding (perhaps because it's very frequent) will have high dot products with everything.

The fix: Always use cosine similarity (normalized dot product): $\text{similarity} = \frac{\mathbf{v}_1 \cdot \mathbf{v}_2}{||\mathbf{v}_1|| \cdot ||\mathbf{v}_2|}$ This measures angle between vectors, independent of magnitude.

Now here's the really cool part: you can do math with meanings! If you go from the dot for "king" to the dot for "man", you're moving in the "this is a male person" direction. If you then go the opposite way to "woman", you'll end up near "queen"! It's like meaning has directions you can walk in.

How do we build this map? We read a LOT of sentences (like, millions) and notice which words appear near each other. If "dog" and "bark" appear in the same sentences often, we put them close on the map. If "dog" and "telephone" never appear together, we put them far apart. The computer figures out where to put every word so that the map makes sense with all those patterns.

That's a word embedding—a map where word positions tell you what they mean!

Skip-gram = Skip the target, get the neighbors Predict context from middle word (looks "outward")

GloVe = Global Vectors Uses Global co-occurrence Vectors (not local windows)

Connections

• One-hot encoding ← what embedings replace
• Cosine similarity ← measuring semantic similarity
• Softmax function ← converting scores to probabilities
• Backpropagation ← how embedings are trained
• Transfer learning ← using pre-trained embeddings
• LSTM / GRU ← often use word embeddings as input
• Attention mechanism ← can replace fixed embedings with context-dependent ones
• BERT ← modern contextualized embeddings
• Subword tokenization ← handles rare words and morphology
• t-SNE ← visualizing high-dimensional embeddings in 2D

#flashcards/ai-ml

What are word embeddings? :: Dense, low-dimensional vector representations of words that capture semantic meaning and relationships in continuous space (typically 50-300 dimensions).

Write the negative sampling loss function for one training example :: $\mathcal{L} = -\log \sigma(\mathbf{u}_{w_t}^T \mathbf{v}_c) - \sum_{i=1}^{k} \log \sigma(-\mathbf{u}_{w_i}^T \mathbf{v}_c)$ where $w_t$ is target, $w_c$ is context, $w_i$ are k negative samples.

Write the GloVe objective function :: $\mathcal{L} = \sum_{i,j=1}^{|V|} f(X_{ij}) ( \mathbf{w}_i^T \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j - \log X_{ij} )^2$ where $X_{ij}$ is co-occurrence count and $f$ is a weighting function.

What is the typical dimensionality of word embeddings? :: 50-300 dimensions, balancing expressiveness and computational efficiency. Common choices: 100, 200, 300.

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)