Embedding layers and tied weights: Foundations

Before you can read the parent note, you need to be able to hear each symbol as a plain sentence and see the picture behind it. This page builds all of them, from nothing, in the order they depend on each other.

1. A "token" and its ID — a number that names a word

Text is messy. Computers only handle numbers. So the very first thing any language model does is chop text into small pieces (called tokens) and hand each distinct piece a whole-number name.

We write a token ID as $x$ . It is one of finitely many whole numbers, starting at $0$ — say $0, 1, 2, \dots$ up to one less than the total number of tokens. (We give that total a name, $V$ , in the very next section, and then write the set properly.)

The picture: a row of numbered lockers. (How text becomes these pieces is the job of 4.2.01-Tokenization-fundamentals and 4.2.04-Subword-tokenization-BPE-WordPiece — we take the finished nametags as given.)

Figure — Embedding layers and tied weights

Why the topic needs it: every symbol later (rows, vectors, logits) is indexed by this locker number. It is the address of everything.

2. $V$ — the vocabulary size (how many lockers)

Now that $V$ has a meaning, we can write the set of every legal token ID cleanly: $x \in {0, 1, 2, \dots, V - 1} .$

Tiny toy: $V = 4$ (just cat, dog, the, ran).
GPT-2 small: $V = 50257$ .

Why the topic needs it: $V$ is one side of the giant embedding table, and it is why the table is huge — half a model's parameters can live here.

3. A vector, and $R^{d}$ — a "meaning code" of $d$ numbers

Now the key idea. Instead of leaving a word as a lonely locker number, we store inside its locker a short list of real numbers. That list is a vector.

The picture: a vector of length $d = 2$ or $3$ is literally an arrow pointing in space. $[0.3, 0.7]$ is an arrow reaching right $0.3$ and up $0.7$ . Longer lists ( $d = 768$ ) are arrows in spaces we can't draw, but the idea stays: direction = meaning. Words that point the same way mean similar things.

Why the topic needs it: the whole point of an embedding is to turn a meaningless locker number into a meaningful direction. Similarity of meaning becomes closeness of arrows.

4. A matrix, and $E \in R^{V \times d}$ — the whole cabinet

Stack one meaning-vector for every token, one per row, and you get a grid of numbers: a matrix.

Bold $E$ (capital) = the whole table. Bold $e_{i}$ (lowercase, subscript) = one row of it. Subscript = "which one".

Why the topic needs it: this single table is the embedding layer. Looking up token $x$ means "go to row $x$ of $E$ ."

5. Indexing $E [x]$ vs. one-hot $o_{x}$ — two views of the same lookup

The parent note shows the lookup twice: once as fast indexing, once as a slow-but-revealing multiplication. Here is why both matter.

The picture: a light switch panel with $V$ switches, exactly one flipped ON.

Multiplying $o_{x}^{⊤} E$ (the little $⊤$ means "lay the column on its side to make a row") walks across the table, multiplies every row by its switch, and keeps only the ON row — giving back $e_{x}$ . So:

$o_{x}^{⊤} E = e_{x} = E [x] .$

Why show the slow way? It proves indexing is secretly a matrix multiply, which is why the same $E$ can later serve as an output layer. Why use the fast way? Multiplying by $V - 1$ zeros is wasted work; jumping straight to row $x$ is instant.

6. The context vector $h_{L}$ — the transformer's "what comes next?" arrow

Before we can talk about scoring the next word, we need a name for what the model hands us after it has read the sentence so far.

The picture: $h_{L}$ is one more arrow living in the same space as the token arrows $e_{i}$ from section 3 — which is exactly why we will be able to compare it against them.

Why the topic needs it: weight tying works by comparing this context arrow to every token's arrow. So we must have it in hand before we can score anything.

7. The dot product $a \cdot b$ — a similarity meter

Weight tying scores the next word by comparing $h_{L}$ to each token arrow. We need one number that measures "how aligned are two arrows?" — that number is the dot product.

Why this tool and not another? We need one number that answers "do these two arrows point the same way?" The dot product does exactly that: it is large-positive when arrows align, zero when they are perpendicular, negative when they oppose. It is the natural "alignment meter" — precisely the question the output layer asks ("which token vector aligns with my context $h_{L}$ ?").

8. Logits $z_{j}$ and bias $b_{j}$ — turning alignment into a raw score

Now we can build the raw score the output layer produces for each candidate token. First name the two ingredients.

Now assemble them. Both symbols on the right ( $h_{L}$ from section 6, the dot product from section 7, $e_{j}$ from section 3, $b_{j}$ just above) are already defined, so we may finally write: $z_{j} = h_{L} \cdot e_{j} + b_{j} .$

So $z_{j}$ is a dot product — the alignment between the context arrow $h_{L}$ and each candidate token's arrow $e_{j}$ — with the token's baseline $b_{j}$ added on top. High alignment → high logit → (after the next step) high probability.

9. Transpose $E^{⊤}$ — flipping the cabinet on its side

Computing $z_{j}$ one token at a time is fine, but we want all $V$ logits at once. The transpose lets one matrix multiply do that.

The picture: rotate the grid 90°; what was horizontal is now vertical.

Why the topic needs it: tying sets the output weight $W_{out} = E^{⊤}$ . Multiplying the context by $E^{⊤}$ computes the dot product of $h_{L}$ with every row of $E$ at once — producing the whole logit vector $z$ in one step. Transpose is the bookkeeping that lets one table do two jobs.

10. Softmax and $P (\cdot)$ — from raw scores to a probability guess

Logits are raw numbers; we want probabilities that are positive and add to $1$ . That conversion is softmax.

Why the exponential $e^{z_{j}}$ ? We need every score to become positive (probabilities can't be negative) and we want bigger logits to win by a lot. Raising $e$ to the power of each logit does both: it is always positive, and it grows fast, so a slightly higher logit gets a noticeably higher share. Dividing by the total forces them to sum to $1$ .

$P (x_{t + 1} ∣ x_{\leq t})$ reads: "the probability of the next token, given everything up to now." The vertical bar $∣$ means "given". This connects to 5.1.03-Cross-entropy-loss, which measures how good these probabilities are.

11. Gradient $\frac{\partial L}{\partial E}$ — the learning nudge

The picture: standing on a hillside ( $L$ is your height), the gradient points uphill; learning steps the opposite way, downhill toward low loss.

Why the topic needs it: with tying, $E$ appears in two places, so two gradient signals add up onto the same table — the "richer supervision" the parent note mentions.

The prerequisite map

Everything upstream feeds the parent topic Embedding layers and tied weights. Position information joins later via 4.3.02-Positional-encoding; the context vector $h_{L}$ is built by 4.4.03-Self-attention-mechanism; parameter savings from tying matter for 6.2.04-Parameter-efficient-fine-tuning.

Equipment checklist

What is a token ID and why is its numeric value meaningless? ::: An integer nametag for a text piece; it's only an address (locker number), not a magnitude — $42$ isn't "more" than $17$ .
What does $R^{d}$ mean in plain words? ::: A list of $d$ real numbers — equivalently, an arrow in $d$ -dimensional space.
What are the two dimensions of $E$ and what does each row mean? ::: $V$ rows (one per token) by $d$ columns; row $i$ is token $i$ 's meaning-vector $e_{i}$ .
Why is indexing $E [x]$ the same as $o_{x}^{⊤} E$ ? ::: The one-hot vector selects exactly row $x$ , zeroing all others, so both return $e_{x}$ ; indexing just skips the wasted zero-multiplies.
What is the context vector $h_{L}$ ? ::: The arrow the transformer produces after reading everything so far, living in $R^{d}$ and summarising "what should come next?"
What single question does the dot product $h_{L} \cdot e_{j}$ answer? ::: How aligned (similar in direction) the context arrow is with token $j$ 's embedding.
What is the bias $b_{j}$ and where does it enter? ::: A per-token base offset added to the logit: $z_{j} = h_{L} \cdot e_{j} + b_{j}$ ; it is a token's context-independent "base popularity".
What does the transpose $E^{⊤}$ do, and why does tying need it? ::: It flips rows to columns ( $V \times d \to d \times V$ ) so one multiply scores the context against every token at once.
Why does softmax use $e^{z_{j}}$ ? ::: To make every score positive and let higher logits win by a large margin; dividing by the total makes them sum to $1$ .
Why does tying give "richer supervision"? ::: $E$ is used at input and output, so gradients from both paths add onto the same table.