Word embeddings (Word2Vec, GloVe): Question bank

This is a rapid-fire bank of conceptual questions for Word embeddings (Word2Vec, GloVe). Each line is a question ::: answer reveal — cover the right side, answer out loud, then check. The goal is not arithmetic (see the computation banks for that) but killing the misconceptions this topic loves to plant.

True or false — justify

A word embedding is a lookup table of learned vectors, one per vocabulary word

True. The "embedding matrix" is exactly a table with one dense row per word; training just adjusts those rows.

One-hot vectors already encode that "cat" and "dog" are similar

False. In One-hot encoding every pair of distinct words is equidistant, so no similarity information exists at all.

Embeddings must have exactly 300 dimensions to work

False. Dimension is a hyperparameter (50–300 are common); more dimensions can capture finer structure but cost more and risk overfitting.

CBOW and Skip-gram learn the same kind of vectors, just with input and output roles swapped

True. Both learn dense context-predicting vectors; CBOW predicts a word from context, Skip-gram predicts context from a word.

Negative sampling computes the exact softmax probability, only faster

False. It replaces the softmax with a different (binary logistic) objective — it approximates the goal, it does not compute the true normalized probability.

GloVe never looks at individual sentences during training

True. GloVe trains on the pre-built global co-occurrence matrix

X

, so it consumes counts, not raw running text.

Two words with high Cosine similarity in embedding space must appear next to each other in text

False. They appear in similar contexts (e.g. "big"/"large"), which need not mean they co-occur — they can be near-synonyms that rarely sit side by side.

The Softmax function denominator in Word2Vec sums over the whole vocabulary

True, and that is exactly why it is expensive — every step touches all

∣ V ∣

output vectors unless you use a shortcut.

Every word in Word2Vec has two vectors during training

True. There is an input embedding

v_{w}

and an output embedding

u_{w}

; final embeddings often use

v_{w}

or the average.

GloVe's weighting

f (x)

is optional cosmetic tuning

False. Without it,

lo g X_{ij}

blows up for

X_{ij} = 0

(log of zero) and frequent pairs dominate the loss —

f

is structurally necessary.

Spot the error

"CBOW predicts the context words given the target word."

Reversed. CBOW predicts the target from context; Skip-gram predicts context from the target.

"We should raise unigram frequencies to the power 1.0 so negative sampling matches real frequencies."

The exponent is

0.75

, not

1.0

. Raising to

0.75

lifts rare words' sampling odds so common words don't monopolize the negatives.

"Because softmax is

O (∣ V ∣)

, hierarchical softmax makes it

O (∣ V ∣/2)

Hierarchical softmax is

O (lo g ∣ V ∣)

— you walk a binary tree of depth

lo g_{2} ∣ V ∣

, not half the vocabulary.

"GloVe fits the dot product to the raw co-occurrence count

X_{ij}

It fits to

lo g X_{ij}

(offset by biases), not the raw count; the log comes from the ratio-to-exponential derivation.

"In Skip-gram, the hidden layer averages all the context embeddings."

That is CBOW. Skip-gram's hidden layer is just the single target embedding

v_{w_{t}}

— no averaging.

"Negative sampling maximizes

σ (u_{w_{i}}^{T} v_{c})

for the sampled negative words

w_{i}

It maximizes

σ (- u_{w_{i}}^{T} v_{c})

for negatives — we want fake pairs scored low, so we push their (negated) score up.

"The GloVe weighting

f (x) = 1

for all frequent pairs, so the most frequent pair contributes the most loss."

The cap at

1

prevents frequent pairs from dominating — it stops their weight growing without bound, equalizing their influence.

"Word2Vec uses Backpropagation only for the softmax version, not for negative sampling."

Both are trained by gradient descent with backprop; negative sampling just changes the loss the gradients flow from.

Why questions

Why do we take the log of the corpus probability in the Word2Vec loss?

A product of many probabilities underflows to zero and is hard to differentiate;

lo g

turns the product into a sum of stable, separately-differentiable terms.

Why does Skip-gram often beat CBOW on rare words?

Each target word spawns one training example per context slot, so a rare word gets several gradient updates per occurrence instead of being averaged away as in CBOW.

Why does GloVe care about ratios of co-occurrence probabilities, not the probabilities themselves?

The raw probability of a context word is dominated by that word's overall frequency; the ratio cancels common frequency and isolates which of two words the context truly distinguishes (e.g. "solid" separates "ice" from "steam").

Why does negative sampling turn a

∣ V ∣

-way softmax into a binary classification?

Distinguishing "real pair vs. a few sampled fakes" only needs a handful of logistic decisions, cutting cost from

O (∣ V ∣)

O (k)

while still shaping the vectors correctly.

Why can we do "king − man + woman ≈ queen" arithmetic on embeddings?

Consistent context patterns make certain semantic differences (like gender) point in roughly the same direction in space, so adding/subtracting those direction vectors moves you to the analogous word.

Why measure closeness with Cosine similarity rather than plain Euclidean distance?

Cosine compares direction (meaning), ignoring vector length, which often reflects word frequency rather than semantics.

Why are embeddings a natural target for Transfer learning?

They are pre-trained on huge corpora to capture general meaning, so a downstream task can reuse them and learn from far less labeled data.

Why does GloVe split each word into a word vector

w_{i}

and a context vector

\tilde{w}_{j}

The co-occurrence matrix would be symmetric otherwise; two distinct vectors break that symmetry and give the optimization more freedom (final embeddings often sum the two).

Edge cases

What does the model do with a word pair that never co-occurs in GloVe?

f (0) = 0

zeroes its loss term entirely, so those absent pairs never contribute — avoiding

lo g 0 = - \infty

What happens to a word never seen in training when you deploy the embeddings?

It has no learned row (out-of-vocabulary); classic Word2Vec/GloVe can't embed it, which is one motivation for Subword tokenization.

What does an embedding for an extremely frequent stopword like "the" tend to look like?

It appears in nearly every context, so its vector is weakly informative and often sits centrally; GloVe's cap and Word2Vec's subsampling deliberately downweight such words.

If the context window size is set to 1, what changes conceptually?

Only the immediately adjacent word counts as context, giving very local, syntax-flavored embeddings; larger windows capture broader, more topical similarity.

Can two different words end up with identical embedding vectors?

Practically no — they'd need identical context statistics; even near-synonyms differ slightly, so vectors are distinct but close.

Why can't you just read a word's meaning off a single embedding dimension?

Meaning is distributed across all dimensions with no built-in interpretability; that opacity is why we project with tools like t-SNE to see structure.

Are these static embeddings context-sensitive — does "bank" get one vector or two?

One fixed vector per word (static). It cannot distinguish river-bank from money-bank; contextual models like BERT fix this by producing per-sentence vectors.

Recall Fast self-check

Skip-gram hidden layer averages context vectors? ::: No — that's CBOW; Skip-gram uses the single target vector. Negative sampling exponent on unigram frequencies? ::: $0.75$ . GloVe fits the dot product to which quantity? ::: $lo g X_{ij}$ (offset by biases). Why cosine over Euclidean? ::: It compares direction/meaning and ignores length/frequency.