Foundations — Causal language modeling objective

Before you can read the parent note without stumbling, you need to own every symbol it throws at you. This page builds them one at a time, from nothing, each with a picture. If you have never seen a subscript, a $\log$, or a $\sum$ before — start here at line one.

1. Tokens and the sequence $\mathbf{x}$

Before we can predict "the next word", we must decide what a "word" even is to a machine. A model cannot eat letters or spaces directly. Instead, text is chopped into small pieces called tokens — sometimes whole words, sometimes word-fragments, sometimes punctuation.

We line these tiles up in order and give the whole line a name: the bold letter $\mathbf{x}$.

Why do we need position numbers at all? Because "I love ML" and "ML love I" contain the same tiles but mean different things. The order is the information. The subscript is how we pin each tile to its slot.

See Transformer Decoder for the machine that actually processes this sequence, and Autoregressive Models for the family this whole idea belongs to.

2. Probability $P$, and conditional probability $P(\cdot \mid \cdot )$

The parent note is drowning in the letter $P$. Here is exactly what it means.

For language: $P(x_2=\text{"love"})$ asks "how likely is the second word to be 'love'?" — before we've looked at anything else.

But language depends on context. "love" is likely after "I", unlikely after "quantum". To express "likely given we already know something", we use the vertical bar $\mid$, read as "given".

Why does the topic need conditionals? Because the entire trick of language modeling is: don't guess words in a vacuum — guess each word conditioned on the words before it. That bar is the heart of the whole objective.

3. The context $x_{<t}$ — "everything before now"

The parent writes $x_{<t}$ constantly. Decode it piece by piece:

• $x$ = a token,
• the little $<t$ below = "at a position less than $t$".

Why "causal"? In the real world, causes come before effects — you can't be influenced by tomorrow. Forcing predictions to depend only on the past mirrors that arrow of time. This is the crucial difference from Masked Language Modeling, which is allowed to look both ways.

So $P_{\theta }(x_t\mid x_{<t})$ is the whole game in one phrase: "probability of the next tile, given all tiles so far."

4. The parameters $\theta$ — the model's tunable knobs

The subscript $\theta$ (Greek letter "theta") appears on every $P_{\theta }$.

$P_{\theta }$ therefore means "the probability according to this particular setting of the knobs". Training = nudging the sliders $\theta$ until the model's guesses match real text. The subscript reminds us: this is the model's opinion, not ground truth.

5. The product $\prod$ and the chain rule

To score a whole sentence, the parent multiplies many conditional probabilities together. The tall symbol for "multiply a list" is $\prod$ (capital Greek "pi").

Now the parent's key formula: $P(\mathbf{x})=P(x_1)\cdot P(x_2\mid x_1)\cdot P(x_3\mid x_1,x_2)\cdots$

Why does the topic need it? Because it turns the impossible task "score an entire sentence at once" into a repeated, manageable task "score one next word at a time". That repetition is what a Transformer Decoder is built to do in parallel.

6. The logarithm $\log$ — turning products into sums

Multiplying hundreds of tiny probabilities (each less than 1) gives an absurdly small number — computers round it to zero (underflow). The fix is a mathematical tool called the logarithm.

Why this tool and not another? We specifically need something that (a) converts a fragile product into a stable sum, and (b) is monotonic — bigger input still gives bigger output, so maximizing the log is the same as maximizing the original. The logarithm is the unique everyday function with both properties. Nothing else lets us keep the answer while dodging underflow.

7. The sum $\sum$ and the negative log-likelihood

Apply the log to the product from Section 5, and every $\times$ becomes a $+$. The "add up a list" symbol is $\sum$ (capital Greek "sigma").

Putting it together gives the parent's core loss: ${\mathcal{L}}_{\text{CLM}}=-{\sum }_{t=1}^T\log P_{\theta }(x_t\mid x_{<t})$

Two last symbols to name:

• $\mathcal{L}$ (fancy script "L") = the loss, a single number measuring how bad the model is. Lower is better; a perfect model ($P=1$ everywhere) scores $\mathcal{L}=0$.
• The leading minus sign flips log's negatives into positives, so "worse guess" = "bigger loss". Minimizing this is identical to maximizing the sentence's probability. This quantity has a name from information theory: Cross-Entropy Loss.

8. Softmax and the vocabulary $|V|$

The model doesn't output a probability directly. It first outputs a raw score for every possible token. The full list of possible tokens is the vocabulary, and its size is written $|V|$ (the bars mean "count of items in the set $V$"). For GPT-2, $|V|=50257$.

The raw scores are called logits, ${\mathbf{z}}_t\in {\mathbb{R}}^{|V|}$ — one real number per vocabulary entry ($\mathbb{R}$ just means "any real number", the superscript $|V|$ means "a list of that many of them").

Why $\exp$ and not just clamp negatives to zero? Because $\exp$ is smooth and always positive, it keeps every token in the running (none is ever hard-zeroed) while still sharply favouring high scores — ideal for the gentle nudging that training needs. Once you have this distribution, Sampling Strategies decide which token to actually pick at generation time.

Prerequisite map

Everything on the left is a raw ingredient; they flow rightward into the single objective the parent topic optimizes.

Equipment checklist

Test yourself — reveal only after you've answered aloud.