Word embeddings (Word2Vec, GloVe)

3.5.11 · AI-ML › Sequence Models

Overview

Word embeddings dense, low-dimensional vector representations hain words ki, jo continuous space mein semantic meaning aur relationships capture karte hain. One-hot encodings (sparse, high-dimensional, koi semantic info nahi) ke unlike, embeddings similar words ko vector space mein paas-paas rakhte hain.

Why Word Embeddings?

One-hot encoding ke saath problem:

• 50,000 words ki vocabulary → 50,000-dimensional vectors
• Har word har doosre word se equally door (koi semantic similarity nahi)
• "cat" aur "dog" utne hi different hain jitne "cat" aur "democracy"

Embeddings kya fix karte hain:

1. Dense representation: 50,000 ki jagah 50-300 dimensions
2. Semantic similarity: Similar words ke similar vectors hote hain (cosine similarity se measure hota hai)
3. Compositionality: Vector operations meaning relationships capture karte hain
4. Transferability: Large corpus par pre-train karo, downstream tasks mein use karo

Word2Vec: Learning from Context

Word2Vec ke do architectures hain:

1. CBOW (Continuous Bag of Words)

CBOW apne surrounding context words se ek target word predict karta hai.

Ye kaise kaam karta hai:

1. Input Layer: Context words ke liye one-hot vectors

2. Hidden Layer: Context word embeddings ka average nikalo $\mathbf{h}=\frac{1}{C}{\sum }_{i=-C,i\ne 0}^C{\mathbf{v}}_{w_{t+i}}$ jahan $C$ context window size hai, ${\mathbf{v}}_w$ word $w$ ka embedding hai

3. Output Layer: Saare vocabulary words ke liye scores compute karo $\text{score}(w)={\mathbf{u}}_w^T\mathbf{h}$ jahan ${\mathbf{u}}_w$ word $w$ ka output embedding hai

4. Softmax: Scores ko probabilities mein convert karo $P(w_t|\text{context})=\frac{\exp ({\mathbf{u}}_{w_t}^T\mathbf{h})}{{\sum }_{w\in V}\exp ({\mathbf{u}}_w^T\mathbf{h})}$

Ye kyu kaam karta hai: Jo words similar contexts mein appear hote hain unhe similar embedding vectors ki taraf push kiya jaata hai.

First principles se derivation:

• Shuru karo goal se: corpus observe karne ki probability maximize karo
• Corpus probability: $P(\text{corpus})={\prod }_{t=1}^{T}P(w_t|{\text{context}}_t)$
• Log lo (product ko sum mein convert karo): $\log P(\text{corpus})={\sum }_{t=1}^T\log P(w_t|{\text{context}}_t)$
• Negative log-likelihood minimize karo (likelihood maximize karne ke equivalent hai)
• Stability ke liye corpus length $T$ se normalize karo

2. Skip-gram

Skip-gram ulta karta hai: ek target word diya hua ho toh context words predict karta hai.

Ye kaise kaam karta hai:

1. Input: Target word $w_t$ ke liye one-hot vector
2. Hidden Layer: Sirf target word embedding ${\mathbf{v}}_{w_t}$ (koi averaging nahi)
3. Output: Har context word ko independently predict karo $P(w_{t+j}|w_t)=\frac{\exp ({\mathbf{u}}_{w_{t+j}}^T{\mathbf{v}}_{w_t})}{{\sum }_{w\in V}\exp ({\mathbf{u}}_w^T{\mathbf{v}}_{w_t})}$

Skip-gram aksar better kyu kaam karta hai: Har target word multiple training examples generate karta hai (ek har context position ke liye), isliye rare words ko zyada updates milte hain.

Computational Challenge: Softmax is Expensive

Denominator ${\sum }_{w\in V}\exp ({\mathbf{u}}_w^T\mathbf{h})$ ko saare vocabulary words (often 50,000+) ke liye scores compute karne padte hain har training step par.

Solution 1: Hierarchical Softmax

Flat softmax ko ek binary tree (Huffman tree) se replace karo:

• Har word ek leaf hai
• Ek word predict karne ke liye, tree traverse karo (|V| comparisons ki jagah log₂|V| binary decisions)
• Ek word ki probability = uske path ke saath probabilities ka product

Complexity: $O(\log |V|)$ ki jagah $O(|V|)$

Solution 2: Negative Sampling

Negative sampling problem ko binary classification mein badal deta hai: "Kya ye word-context pair real hai ya fake?"

Trick:

• Har real (target, context) pair ke liye, $k$ random "negative" words sample karo (typically 5-20)
• Logistic regression use karke real pair ko negative pairs se distinguish karna seekho

Derivation:

• $\sigma ({\mathbf{u}}_{w_t}^T{\mathbf{v}}_c)$ = probability ki real pair real hai (high honi chahiye)
• $\sigma (-{\mathbf{u}}_{w_i}^T{\mathbf{v}}_c)=1-\sigma ({\mathbf{u}}_{w_i}^T{\mathbf{v}}_c)$ = probability ki fake pair fake hai (high honi chahiye)
• Negative samples noise distribution $P_n(w)\propto f(w{)}^{0.75}$ se draw kiye jaate hain (smoothed unigram)
  • $0.75$ kyu? Frequent vs. rare words ko balance karta hai—smoothing ke bina, sirf common words sample honge

Complexity: $O(k)$ jahan $k\ll |V|$

GloVe: Global Vectors for Word Representation

Word2Vec local hai (chhote context windows dekhta hai). GloVe global hai (corpus-wide co-occurrence statistics use karta hai).

"Ice" aur "steam" words socho, aur context words "solid" vs. "gas":

• $P(\text{solid}|\text{ice})$ high hai, $P(\text{solid}|\text{steam})$ low hai → ratio bada hai
• $P(\text{gas}|\text{ice})$ low hai, $P(\text{gas}|\text{steam})$ high hai → ratio chota hai
• "Fashion" jaise unrelated words ke liye, dono probabilities low hain → ratio ≈ 1

Idea: Word vectors ko ye ratios encode karne chahiye.

GloVe Model

Define karo:

• $X_i={\sum }_k{X}_{ik}$ = word $i$ ke context mein saare words ka total count
• $P_{ij}=P(j|i)=X_{ij}/X_i$ = probability ki word $j$, word $i$ ke context mein appear hoga

Goal: Aise embedding vectors dhundho jinka dot product log co-occurrence se relate kare.

Components explained:

• ${\mathbf{w}}_i$: word $i$ ke liye word embedding
• $\widetilde{{\mathbf{w}}_j}$: word $j$ ke liye context embedding
• $b_i,{\tilde{b}}_j$: har word ke liye bias terms
• $f(X_{ij})$: zero aur rare co-occurrences handle karne ke liye weighting function

Ye weighting kyu?

• $f(0)=0$: Aise word pairs par train mat karo jo kabhi co-occur nahi hue (log(0) ke saath infinite loss contribute karte)
• $f(x)<1$ rare pairs ke liye: Rare co-occurrences noisy hoti hain, unhe downweight karo
• $f(x)=1$ frequent pairs ke liye: Weight ko cap karo taaki bahut frequent pairs dominate na karein

Ratio insight se derivation:

Shuru karo goal se: ${\mathbf{w}}_i^T{\mathbf{w}}_j$ ko $P_{ij}/P_{ik}$ ratios se relate hona chahiye.

1. Ratio form: $F({\mathbf{w}}_i,{\mathbf{w}}_j,{\tilde{\mathbf{w}}}_k)=P_{ij}/P_{ik}$ jahan $F$ koi function hai
2. Kyunki hum vectors chahte hain, dot products use karo: $F({\mathbf{w}}_i^T{\mathbf{w}}_j-{\mathbf{w}}_i^T{\mathbf{w}}_k)=P_{ij}/P_{ik}$
3. Homomorphism: $F(a-b)=F(a)/F(b)$ suggest karta hai $F=\exp$
4. Toh: $\exp ({\mathbf{w}}_i^T{\mathbf{w}}_j)=P_{ij}{X}_{ij}/X_i$
5. Log lo: ${\mathbf{w}}_i^T{\mathbf{w}}_j=\log X_{ij}-\log X_i$
6. $\log X_i$ ko bias mein absorb karo: ${\mathbf{w}}_i^T{\mathbf{w}}_j+b_i+b_j=\log X_{ij}$
7. Word vs. context embeddings alag karo (training ke liye symmetry todta hai): ${\mathbf{w}}_i^T{\tilde{\mathbf{w}}}_j+b_i+{\tilde{b}}_j=\log X_{ij}$
8. Weighting ke saath squared error minimize karo: upar wala loss function

Example 1: CBOW Forward Pass

Sentence: "The cat sat on the mat" Task: "sat" predict karo context ["cat", "on"] se (window size = 1) Vocabulary: {the, cat, sat, on, mat} (size 5) Embedding dimension: 3

Step 1: Context embeddings lo

Embedding matrix E (5 × 3):
"cat" → [0.2, 0.5, 0.1]
"on"  → [0.3, 0.4, 0.2]

Step 2: Context embeddings ka average nikalo $\mathbf{h}=\frac{1}{2}([0.2,0.5,0.1]+[0.3,0.4,0.2])=[0.25,0.45,0.15]$ Average kyu? Context ka "central meaning" represent karta hai, order-invariant.

Step 3: Output scores compute karo

Output matrix W_out (3 × 5):
Scores = W_out^T · h
"the": [0.1, 0.2, 0.3]^T · [0.25, 0.45, 0.15] = 0.025 + 0.09 + 0.045 = 0.16
"cat": [...]^T · h = 0.21
"sat": [...]^T · h = 0.42  ← highest score
"on":  [...]^T · h = 0.18
"mat": [...]^T · h = 0.14

Dot product kyu? Context representation aur candidate word ke beech alignment measure karta hai.

Step 4: Softmax $P(\text{sat}|\text{context})=\frac{\exp (0.42)}{\exp (0.16)+\exp (0.21)+\exp (0.42)+\exp (0.18+\exp (0.14)}=\frac{1.52}{1.17+1.23+1.52+1.20+1.15}=\frac{1.52}{6.27}\approx 0.24$

Softmax kyu? Scores ko valid probability distribution mein convert karta hai (sum 1 hota hai).

Example 2: Negative Sampling

Positive pair: (target="cat", context="sat") Task: $k=2$ negatives ke saath negative sampling use karke embeddings train karo

Step 1: Positive score compute karo

v_cat = [0.2, 0.5, 0.1]  (target embedding)
u_sat = [0.3, 0.4, 0.2]  (context embedding)
score_pos = v_cat · u_sat = 0.06+ 0.20 + 0.02 = 0.28

Step 2: Negative words sample karo Context mein nahi hain aise 2 random words sample karo: "the", "mat"

u_the = [0.1, 0.2, 0.3]
u_mat = [0.4, 0.1, 0.2]
score_neg1 = v_cat · u_the = 0.02 + 0.10 + 0.03 = 0.15
score_neg2 = v_cat · u_mat = 0.08 + 0.05 + 0.02 = 0.15

Random sampling kyu? Puri vocabulary ka efficient approximation hai.

Step 3: Loss compute karo $\mathcal{L}=-\log \sigma (0.28)-\log \sigma (-0.15)-\log \sigma (-0.15)$ $=-\log (0.57)-\log (0.46)=0.56+0.78+0.78=2.12$ Interpretation: Positive pair ka score high hona chahiye (sigmoid ke baad 1 ke paas), negatives ka low hona chahiye.

Step 4: Gradient update

∂L/∂v_cat pushes v_cat closer to u_sat (positive context)
∂L/∂v_cat pushes v_cat away from u_the, u_mat (negative contexts)

Result: "cat" embedding un words ki taraf move karta hai jo actually uske paas appear hote hain, unrelated words se door.

Example 3: GloVe Training

Mini co-occurrence matrix:

        cat   sat   mat
dog     5     2     3
cat     08     4

$X_{\text{dog,cat}}=5$: "cat" corpus mein "dog" ke context mein 5 baar appear hua.

Training pair: (dog, cat) with $X_{ij}=5$

Current embeddings:

w_dog = [0.5, 0.3]
w̃_cat = [0.4, 0.6]
b_dog = 0.1
b̃_cat = 0.2

Step 1: Prediction compute karo ${\hat{X}}_{ij}=\exp ({\mathbf{w}}_{\text{dog}}^T{\tilde{\mathbf{w}}}_{\text{cat}}+b_{\text{dog}}+{\tilde{b}}_{\text{cat}})$ $=\exp (0.5\cdot 0.4+0.3\cdot 0.6+0.1+0.2)=\exp (0.68)=1.97$ Exponent kyu? Hum log co-occurrence model kar rahe hain, toh exp count scale par wapas convert karta hai.

Step 2: Is pair ke liye loss compute karo $f(5)=(5/100{)}^{0.75}=0.084$ $\text{loss}=0.084\cdot (\log 1.97-\log 5{)}^2=0.084\cdot (0.68-1.61{)}^2=0.084\cdot 0.86=0.072$ Squared error kyu? Simple, differentiable objective jo predicted log-count ko actual log-count ki taraf drive karta hai.

Step 3: Embeddings update karo Gradient ${\mathbf{w}}_{\text{dog}}$ aur ${\tilde{\mathbf{w}}}_{\text{cat}}$ ko push karta hai taaki unka dot product $\log (5)-b_{\text{dog}}-{\tilde{b}}_{\text{cat}}$ ke paas aaye.

Kai updates ke baad: Jo words frequently co-occur hote hain unke high dot products honge, rarely co-occur hone wale words ke low dot products honge.

Properties of Trained Embeddings

Semantic Similarity

Embeddings ke beech cosine similarity meaning capture karti hai: $\text{similarity}({\mathbf{v}}_1,{\mathbf{v}}_2)=\frac{{\mathbf{v}}_1\cdot {\mathbf{v}}_2}{||{\mathbf{v}}_1||\cdot ||{\mathbf{v}}_2|}$

Similar words ki similarity 1 ke paas hoti hai:

• cos(king, queen) ≈ 0.8
• cos(cat, dog) ≈ 0.76

Analogies

Vector arithmetic relationships capture karta hai: ${\mathbf{v}}_{\text{king}}-{\mathbf{v}}_{\text{man}}+{\mathbf{v}}_{\text{woman}}\approx {\mathbf{v}}_{\text{queen}}$

Ye kyu kaam karta hai:

• ${\mathbf{v}}_{\text{king}}-{\mathbf{v}}_{\text{man}}$ = "royalty" direction ("maleness" remove karta hai)
• ${\mathbf{v}}_{\text{woman}}$ add karna female royalty par project karta hai

Doosre examples:

• Paris - France + Italy ≈ Rome (capital of...)
• walking - walked + saw ≈ seeing (verb tense)

Clustering

Words embedding space mein semantic category ke hisaab se cluster karte hain:

• Animals saath cluster karte hain
• Countries saath cluster karte hain
• Verbs aspect/tense ke hisaab se cluster karte hain

Word2Vec vs. GloVe

Practice mein: Dono high-quality embeddings produce karte hain. Word2Vec (especially Skip-gram with negative sampling) speed ki wajah se zyada popular hai. GloVe prefer hota hai jab global statistics par explicit control chahiye ho.

Galti: "'bank' (financial) aur 'bank' (river) same word hain, toh unka same embedding hoga. Perfect!"

Kyu sahi lagta hai: Ek word = ek vector natural lagta hai.

Problem: Word2Vec aur GloVe har word type ke liye ek vector seekhte hain, un saare contexts ka average lekar jahan wo word appear hota hai. Polysemous words (multiple meanings) ke muddy embeddings ho jaate hain.

Fix:

• Contextualized embeddings use karo (ELMo, BERT) jahan "bank" ko context ke hisaab se alag vectors milte hain
• Ya sense2vec use karo jo word contexts ko alag senses mein cluster karta hai

Galti: Word2Vec ko 10,000 sentences par train karna aur achhe embeddings expect karna.

Kyu sahi lagta hai: "Deep learning chhote data par bhi kaam karta hai agar carefully tune karun."

Problem: Word embeddings ko statistical patterns capture karne ke liye millions of word occurrences chahiye. Chhote data ke saath:

• Rare words ke embeddings unreliable hote hain (few training examples)
• Semantic relationships noisy hoti hain
• Analogies achhe se kaam nahi karti

Fix:

• Pre-trained embeddings use karo (billions of words par trained: Google News, Wikipedia, Common Crawl)
• Agar zaroorat ho toh apne domain par pre-trained embeddings fine-tune karo
• Ya subword embeddings (FastText) use karo jo rare words ko better handle karta hai

Galti: Similarity measure karne ke liye ${\mathbf{v}}_1\cdot {\mathbf{v}}_2$ compute karna.

Kyu sahi lagta hai: "Bada dot product = zyada similar."

Problem: Dot product vector magnitude se affect hota hai, sirf direction se nahi. Ek word jiska high-magnitude embedding hai (shayad kyunki wo bahut frequent hai) har cheez ke saath high dot products rakhega.

Fix: Hamesha cosine similarity use karo (normalized dot product): $\text{similarity}=\frac{{\mathbf{v}}_1\cdot {\mathbf{v}}_2}{||{\mathbf{v}}_1||\cdot ||{\mathbf{v}}_2|}$ Ye vectors ke beech angle measure karta hai, magnitude se independent.

Ab ye bahut cool part hai: tum meanings ke saath math kar sakte ho! Agar "king" dot se "man" dot ki taraf jaao, tum "ye ek male person hai" direction mein move kar rahe ho. Agar phir "woman" ki taraf ulti direction mein jaao, toh tum "queen" ke paas pahunch jaoge! Jaise meaning ke directions hain jin mein tum chal sakte ho.

Hum ye map kaise banate hain? Hum BAHUT saare sentences padhte hain (jaise, millions) aur notice karte hain ki kaun se words ek doosre ke paas appear hote hain. Agar "dog" aur "bark" aksar same sentences mein hain, unhe map par paas rakhte hain. Agar "dog" aur "telephone" kabhi saath nahi aate, unhe door rakhte hain. Computer figure out karta hai har word ko kahan rakhna hai taaki map un saare patterns ke saath sense kare.

Ye hai word embedding—ek map jahan word positions batate hain unka kya matlab hai!

Skip-gram = Skip the target, get the neighbors Beech ke word se context predict karo ("outward" dekhta hai)

GloVe = Global Vectors Global co-occurrence Vectors use karta hai (local windows nahi)

Connections

• One-hot encoding ← jise embeddings replace karti hain
• Cosine similarity ← semantic similarity measure karna
• Softmax function ← scores ko probabilities mein convert karna
• Backpropagation ← embeddings kaise train hoti hain
• Transfer learning ← pre-trained embeddings use karna
• LSTM / GRU ← aksar word embeddings input ke roop mein use karte hain
• Attention mechanism ← fixed embeddings ko context-dependent ones se replace kar sakta hai
• BERT ← modern contextualized embeddings
• Subword tokenization ← rare words aur morphology handle karta hai
• t-SNE ← high-dimensional embeddings ko 2D mein visualize karna

#flashcards/ai-ml

Word embeddings kya hain? :: Dense, low-dimensional vector representations of words jo continuous space mein semantic meaning aur relationships capture karti hain (typically 50-300 dimensions).

Ek training example ke liye negative sampling loss function likhein :: $\mathcal{L}=-\log \sigma ({\mathbf{u}}_{w_t}^T{\mathbf{v}}_c)-{\sum }_{i=1}^k\log \sigma (-{\mathbf{u}}_{w_i}^T{\mathbf{v}}_c)$ jahan $w_t$ target hai, $w_c$ context hai, $w_i$ k negative samples hain.

GloVe objective function likhein :: $\mathcal{L}={\sum }_{i,j=1}^{|V|}f(X_{ij})({\mathbf{w}}_i^T{\tilde{\mathbf{w}}}_j+b_i+{\tilde{b}}_j-\log X_{ij}{)}^2$ jahan $X_{ij}$ co-occurrence count hai aur $f$ weighting function hai.

Word embeddings ki typical dimensionality kya hoti hai? :: 50-300 dimensions, expressiveness aur computational efficiency balance karte hue. Common choices: 100, 200, 300.

Concept Map