Embedding layers and tied weights: Question bank

This is a concept-trap question bank for the parent note. Every item targets a specific misunderstanding — a place where the words sound right but the geometry is wrong. Answer out loud before you reveal. If your reasoning differs from the answer even when your yes/no matches, treat it as a miss.

Before you start, three plain-word reminders so every symbol below is earned:

Prerequisites if any line feels shaky: 4.2.01-Tokenization-fundamentals, 4.2.04-Subword-tokenization-BPE-WordPiece, 4.3.02-Positional-encoding, 4.4.03-Self-attention-mechanism, 5.1.03-Cross-entropy-loss, 6.2.04-Parameter-efficient-fine-tuning.

True or false — justify

The embedding lookup

E [x]

is a matrix multiplication under the hood

False in practice — it is a pure indexing (retrieval) of one row. It is mathematically equivalent to multiplying a one-hot vector by

E

, but computing it that way wastes

V \times d

multiplications against zeros, so real code just fetches the row.

Weight tying makes the input embedding and the output projection the same matrix

False — they are the same table but transposed:

W_{out} = E^{T}

. Input selects a row; output dot-products the hidden state against every row. Same numbers, inverse operations.

Tying weights always reduces model quality because the model has fewer free parameters

False — it often improves generalization. Sharing forces one consistent meaning-space, and for large

V

the freed capacity would mostly have overfit the vocabulary anyway.

The token "cat" gets a different embedding depending on its position in the sentence

False — the token embedding

e_{cat}

is position-independent. Position enters separately via

h_{0} = E [x] + PositionalEncoding (pos)

(see 4.3.02-Positional-encoding).

Embeddings start out already knowing that "cat" and "dog" are similar

False — they start random. Similarity between related tokens is learned through gradient descent driven by the loss (see 5.1.03-Cross-entropy-loss); it is an outcome, not an initialization.

With tying, the embedding matrix receives gradient signal from two places at once

True —

E

appears at the input and (transposed) at the output, so the two gradient paths add:

\frac{\partial L}{\partial E} = \frac{\partial L}{\partial E _{in}} + (\frac{\partial L}{\partial W _{out}})^{T}

. This is richer supervision, not double-counting error.

A larger embedding dimension

d

always means more expressive tokens with no downside

False — bigger

d

raises parameter count (

V d

) and can overfit; expressiveness saturates, and every downstream layer widens too. It is a trade-off, not a free win.

The logit

z_{j} = h_{L} \cdot e_{j}

is a probability

False — it is a raw score. Probabilities come only after softmax normalizes all logits so they are positive and sum to

1

Spot the error

"Because tying halves the embedding+projection parameters, GPT-2 small drops from 77M to about 38.6M total model parameters."

The error is scope: tying halves only the embedding+projection block (≈77M → ≈38.6M), not the whole model. The transformer blocks are unchanged, so the full model shrinks by that fixed amount, not by half overall.

"To embed a token we build its one-hot vector, then multiply by

E

— that's why it costs

O (V \cdot d)

The error is treating the conceptual form as the implementation. We never build the one-hot in practice; direct indexing

E [x]

O (1)

-ish retrieval, not

O (V d)

"Since

W_{out} = E^{T}

, the output layer picks token

j

that is farthest from the context."

The error is the sign of the goal: high

h_{L} \cdot e_{j}

means most aligned (closest in direction), and softmax gives that token the highest probability — closest wins, not farthest.

"Weight tying removes the output bias too, since the whole projection is just

E^{T}

The error is forgetting the bias: tied logits are

z = h_{L} E^{T} + b

. The bias

b \in R^{V}

is a separate small parameter and is usually kept.

"Pretrained embeddings transfer because every model uses the same random seed."

The error is the cause: they transfer because they encode real linguistic structure (semantic/syntactic relationships), learned from data — nothing to do with seeds.

"Input embedding and output projection are the same question asked twice."

The error is calling them the same question. Input asks "what does this token mean?"; output asks "which token comes next?" — inverse questions, which is exactly why sharing (transposed) weights is sensible.

Why questions

Why can the same matrix serve both input and output roles at all?

Because both roles live in the same

d

-dimensional space: a token's meaning arrow and the context arrow can be compared by dot product. One learned geometry answers "represent this" and "score against this."

Why does the output logit use a dot product specifically, rather than, say, Euclidean distance?

The dot product

h_{L} \cdot e_{j}

measures alignment (direction + magnitude) and is exactly the linear operation

h_{L} E^{T}

that a matrix multiply computes for all tokens at once — cheap, differentiable, and softmax-ready.

Why do we separate token embedding from positional information instead of baking position into

E

So "cat" keeps one reusable meaning vector regardless of where it appears. Mixing position in would force a separate embedding per (token, position) pair — vastly more parameters and no shared learning across positions.

Why does tying give "richer supervision" to the embeddings?

Each embedding row now learns from two objectives at once: what tokens mean in context (input path) and what tokens are likely next (output path). Both gradients update the same numbers, so each row is pinned down more tightly.

Why is one-hot times

E

worth teaching if we never compute it?

It reveals that the lookup is linear algebra — which justifies why

E

can also legitimately act as a linear projection at the output, making weight tying principled rather than a hack.

Why might a practitioner still choose not to tie weights?

When input and output distributions genuinely differ (e.g. different input/output vocabularies, or encoder–decoder with distinct scripts), forcing one shared space can hurt; untied gives independent freedom. Also relevant when only tuning adapters (see 6.2.04-Parameter-efficient-fine-tuning).

Edge cases

What happens to

E [x]

if a token ID

x

equals

V

(one past the last row)?

It is out of bounds — rows are indexed

0

V - 1

. This errors (or silently corrupts memory in unsafe code); it's a classic off-by-one from miscounting vocabulary size.

If a token never appears in training, what does its embedding row look like after training?

It stays essentially at its random initialization — no gradient ever flowed to that row, so it learned nothing. This is why rare/unseen tokens behave unpredictably; subword schemes (see 4.2.04-Subword-tokenization-BPE-WordPiece) exist partly to avoid unseen whole-word rows.

d = 1

(one-dimensional embeddings), can the model still distinguish many tokens well?

Barely — every token becomes a single number on a line, so "similar" collapses to "numerically close." There is far too little room to encode independent semantic and syntactic relationships; expressiveness is crippled.

What is the output distribution before any training, with tied random weights?

Roughly uniform-ish noise — logits are dot products of random arrows, so no token is systematically favored. Softmax spreads probability nearly evenly; the model has no linguistic preference yet.

If two tokens are given identical embedding rows, what does the tied output layer do with them?

It produces identical logits for both, so it can never prefer one over the other. They are indistinguishable to the model in both input meaning and output scoring — a degenerate collapse.

If the hidden state

h_{L}

is the zero vector, what are all the tied logits?

Every dot product

0 \cdot e_{j} = 0

, so all logits equal just the biases

b_{j}

. The prediction is driven entirely by the bias term, i.e. the model's prior over tokens with no context contribution.

Recall One-line self-test

Same table, inverse jobs — input selects a row, output dot-products against every row ::: that is weight tying in nine words, and remembering the transpose is the whole trap.