Foundations — Self-supervised pretraining objectives

Before you can read the parent topic with confidence, you need to own every symbol it throws at you. This page builds each one from nothing, in an order where each brick rests on the one below it.

0. What is a "token"? (the atom of everything)

Before any math, we need the thing the math counts. The parent note writes sequences like the, cat, sat. Those pieces are called tokens.

The picture: imagine a sentence as a train. Each carriage is one token. The model never sees letters — it sees a line of carriages, each stamped with an ID number.

Why the topic needs it: every formula in the parent sums over tokens (that is what ${\sum }_{t=1}^T$ means — more on that below). Without knowing the atom, the sum is meaningless.

1. Subscripts and the sequence $x_1,\dots ,x_T$

The parent writes $x_1,\dots ,x_T$. Let's earn every part.

• $x$ is just a name for "a token in our sentence."
• The little number below — the subscript — says which position in the train. $x_1$ is the first carriage, $x_2$ the second.
• $T$ (capital) is the total number of tokens in the sentence. So $x_T$ is the last carriage.
• $x_1,\dots ,x_T$ read aloud: "$x$-one, $x$-two, all the way to $x$-$T$" — the whole train.

Why the topic needs it: causal language modeling predicts $x_t$ from $x_{<t}$. That single subscript trick ("everything before me") is the whole idea of "predict the next word."

2. Probability $p(\cdot )$ — a number between 0 and 1

The heart of the topic is the symbol $p$.

The picture: a probability is a slice of a pie. The whole pie is 1 (100% of belief). If the model thinks the next word is cat with $p=0.5$, it has handed half the pie to cat and split the other half among all other words.

Why not just output one word? Because language is genuinely uncertain — after "the cat" many words are plausible. A single guess throws away that nuance; a distribution over all words keeps it. This is why the model's final layer is a probability list, not a single answer.

The vertical bar: conditional probability $p(A\mid B)$

The parent writes $p_{\theta }(x_t\mid x_{<t})$. The bar | reads "given".

So $p(x_t\mid x_{<t})$ = "how likely is token $x_t$, given we already know all the words before it." That is exactly the next-word guess.

3. The parameters $\theta$ — the model's knobs

You keep seeing $p_{\theta }$ and ${\theta }^{\star }$. What is that Greek letter?

The picture: think of a giant mixing desk with billions of sliders. $\theta$ names the entire configuration of every slider at once. Training = finding the slider positions that make good guesses.

• ${\theta }^{\star }$ (theta-star) = the best possible knob setting — the one we're searching for.
• $\arg {\max }_{\theta }f(\theta )$ reads: "the value of $\theta$ that makes $f$ as large as possible." ($\arg$ = "the argument/input that achieves it," not the max value itself.)

Why the topic needs it: training is literally ${\theta }^{\star }=\arg {\max }_{\theta }(\dots )$ — "find the knob setting that maximizes how well we predict the hidden words." Everything else is detail.

4. Big-$\Sigma$ (sum) and big-$\Pi$ (product)

Two tall symbols do all the "combine over the sentence" work.

The picture: $\Sigma$ is a conveyor belt where each item drops a coin into one shared jar. $\Pi$ is a chain where each link multiplies the running total.

Why both appear: probabilities of independent-ish steps multiply (that's the chain rule, $\Pi$). But multiplying millions of tiny numbers underflows to zero on a computer, so we take a $\log$ (next section) which turns products into sums — hence the loss uses $\Sigma$.

5. The logarithm $\log$ — why it shows up everywhere

The parent's losses are full of $\log p$. Why?

The picture: $-\log p$ is a "surprise meter." When the model gave the correct word high probability ($p$ near 1), surprise is near 0. When it gave the correct word almost no probability ($p$ near 0), surprise shoots up toward infinity — the model was shocked by reality.

Why this exact tool and not, say, squared error? Two reasons the topic relies on: (1) $\log$ converts the chain-rule product into a sum we can average and differentiate; (2) $-\log p$ is precisely the amount of information (in "nats") the true word carries — minimizing average surprise is the same as maximizing likelihood. That's the bridge to Cross-entropy Loss.

6. From surprise to Cross-Entropy and Softmax

Now we can read the loss $\mathcal{L}$.

Why "cross"-entropy? Because we compare two distributions: the truth (a pie with all its weight on one word — a "one-hot" pie) and the model's pie. Cross-entropy measures how many extra bits of surprise the model pays for not matching the truth.

7. Perplexity — surprise, un-logged

The picture: loss lives in "log-land" (nice for math); perplexity drags it back to "word-count land" (nice for humans). $\mathcal{L}=1.04\Rightarrow \text{PPL}=e^{1.04}\approx 2.83$ → "about as confused as choosing among 3 words."

8. Attention & masking — the "no peeking" rule

The parent says a token "may only attend to positions $\le t$." Two words to unpack: attend and mask.

Why the topic needs it: in next-word prediction the answer is the next word. If a token could attend forward, it would literally read the answer — no learning. The mask enforces honesty. (Contrast: BERT/MLM removes the curtain but hides target words with [MASK] instead — same goal, different trick.)

Prerequisite map

Everything flows downward into the parent topic: Self-supervised pretraining objectives, which then feeds into Fine-Tuning LLMs.

Equipment checklist

Test yourself — cover the right side. If any answer is fuzzy, re-read that section before the parent note.