Causal language modeling objective

What Problem Does It Solve?

Language has structure: the word "love" is more likely after "I" than after "quantum". Causal LM learns this conditional probability distribution over sequences:

$P(\text{sentence}) = P(w_1) \cdot P(w_2 | w_1) \cdot P(w_3 | w_1, w_2) \cdots P(w_n | w_1 \ldots w_{n-1})$

Why factorize this way? By the chain rule of probability, any joint distribution can be written as a product of conditionals. The causal constraint means each $P(w_t | w_{<t})$ is a forward-looking prediction, matching how we naturally produce language.

Figure — Causal language modeling objective

Definition Formal Definition

Given a sequence of tokens $\mathbf{x} = (x_1, x_2, \ldots, x_T)$ , the causal language modeling objective is to minimize the negative log-likelihood: $\mathcal{L}_{\text{CLM}} = -\sum_{t=1}^{T} \log P_\theta(x_t \mid x_{<t})$

where:

$x_{<t} = (x_1, \ldots, x_{t-1})$ is the context (all tokens before position $t$ )
$P_\theta(x_t \mid x_{<t})$ is the model's predicted probability for token $x_t$
$\theta$ represents model parameters (transformer weights, embeddings, etc.)

Key constraint: The model architecture (attention masks, positional encodings) ensures $P_\theta(x_t \mid x_{<t})$ cannot access $x_t, x_{t+1}, \ldots, x_T$ .

Derivation from First Principles

Step 1: Start with Maximum Likelihood

We want a model that assigns high probability to real text. For a dataset $\mathcal{D}$ of sequences, maximize:

$\prod_{\mathbf{x} \in \mathcal{D}} P_\theta(\mathbf{x})$

Why product? Assuming independence between training examples.

Step 2: Apply Chain Rule to Each Sequence

For one sequence $\mathbf{x} = (x_1, \ldots, x_T)$ :

$P_\theta(\mathbf{x}) = P_\theta(x_1, x_2, \ldots, x_T) = \prod_{t=1}^{T} P_\theta(x_t \mid x_{<t})$

Why this factorization? The chain rule of probability: $P(A, B) = P(A) P(B|A)$ , extended to $T$ variables. The causal ordering ( $x_1$ first, then $x_2$ given $x_1$ , etc.) is arbitrary but matches text's left-to-right flow.

Step 3: Take Negative Log for Optimization

Maximizing product = minimizing negative log-likelihood:

$\mathcal{L}_{\text{CLM}} = -\log P_\theta(\mathbf{x}) = -\sum_{t=1}^{T} \log P_\theta(x_t \mid x_{<t})$

Why negative log?

Numerical stability: Products of small probabilities underflow; sums of logs don't.
Additivity: Loss over a batch is just the sum of per-example losses.
Convexity: Log-loss is convex for exponential families (like softmax output).

Step 4: Average Over Dataset

$\mathcal{L}_{\text{CLM}}(\mathcal{D}) = -\frac{1}{|\mathcal{D}|} \sum_{\mathbf{x} \in \mathcal{D}} \sum_{t=1}^{T} \log P_\theta(x_t \mid x_{<t})$

This is the cross-entropy loss between the true next-token distribution (one-hot) and the model's predicted distribution.

Formula Implementation Details

In practice:

Model output: Transformer outputs logits $\mathbf{z}_t \in \mathbb{R}^{|V|}$ for each position $t$ , where $|V|$ is vocabulary size.
Softmax to probabilities: $P_\theta(x_t = v \mid x_{<t}) = \frac{\exp(z_{t,v})}{\sum_{v'=1}^{|V|} \exp(z_{t,v'})}$
Cross-entropy loss: $\mathcal{L}_t = -\log P_\theta(x_t \mid x_{<t}) = -z_{t, x_t} + \log \sum_{v'=1}^{|V|} \exp(z_{t,v'})$

Why this form? Since the true distribution is one-hot at $x_t$ , cross-entropy reduces to negative log-probability of the correct class.
Causal masking: In self-attention, mask future positions so $\text{Attention}(Q_t, K, V)$ only attends to keys/values at positions $\leq t$ .

Worked Examples

Example 1: Tiny Sequence

Sequence: "I love ML"
Tokens: $x_1 = \text{I}, x_2 = \text{love}, x_3 = \text{ML}$

Step-by-step loss:

Position 1: $\mathcal{L}_1 = -\log P_\theta(x_1 = \text{I} \mid \varnothing)$ Why no context? $x_1$ is the first token; model predicts from a start-of-sequence token or learns unconditional distribution.
Position 2: $\mathcal{L}_2 = -\log P_\theta(x_2 = \text{love} \mid x_1 = \text{I})$ Why only $x_1$ ? Causal constraint: model sees "I", predicts "love".
Position 3: $\mathcal{L}_3 = -\log P_\theta(x_3 = \text{ML} \mid x_1 = \text{I}, x_2 = \text{love})$ Why both $x_1, x_2$ ? All previous context available.

Total loss: $\mathcal{L}_{\text{CLM}} = \mathcal{L}_1 + \mathcal{L}_2 + \mathcal{L}_3$

Concrete numbers: Suppose model outputs:

$P_\theta(\text{I} \mid \varnothing) = 0.1$ → $\mathcal{L}_1 = -\log(0.1) \approx 2.30$
$P_\theta(\text{love} \mid \text{I}) = 0.4$ → $\mathcal{L}_2 = -\log(0.4) \approx 0.92$
$P_\theta(\text{ML} \mid \text{I love}) = 0.05$ → $\mathcal{L}_3 = -\log(0.05) \approx 3.00$

Total: $2.30 + 0.92 + 3.00 = 6.22$

Why this number? Lower loss = model assigns higher probability to the true sequence. Perfect prediction ( $P=1$ ) gives loss $0$ .

Example 2: Generation vs. Training

Training:

Input: "The cat sat on the"
Targets: "cat sat on the mat" (shifted by 1)
Loss: $-\log P(\text{cat}|\text{The}) - \log P(\text{sat}|\text{The cat}) - \ldots$

Generation (autoregressive sampling):

Start: "The"
Sample $x_2 \sim P_\theta(\cdot | \text{The})$ → get "cat"
Sample $x_3 \sim P_\theta(\cdot | \text{The cat})$ → get "sat"
Continue until end-of-sequence token

Why same objective works for both? Training learns $P_\theta(x_t | x_{<t})$ at every position. Generation repeatedly applies this learned conditional to extend the sequence.

Worked example Real Model (GPT-2 Small)

Architecture: 12-layer transformer, 768-dim embeddings, 50,257 vocab Loss computation per batch:

# Pseudo-code (PyTorch-like)
logits = model(input_ids)  # shape: (batch, seq_len, vocab_size)
shift_logits = logits[:, :-1, :]  # exclude last position
shift_labels = input_ids[:, 1:]   # exclude first position (shift targets)
loss = cross_entropy(
    shift_logits.view(-1, vocab_size),
    shift_labels.view(-1),
    ignore_index=pad_token_id  # don't penalize padding
)

Why shift? Predict position $t$ from positions $< t$ . Input at position $t-1$ predicts label at position $t$ .

Typical loss values:

Random init: $\log(50257) \approx 10.8$ (uniform distribution over vocab)
Trained GPT-2: $\approx 3.5$ on validation set
Overfitted: $< 0.1$ on training set (memorization)

Common mistake Common Mistakes

Mistake 1: "Loss includes future tokens"

Wrong thinking: "Model sees entire sentence, predicts all tokens at once." Why it feels right: Parallel training (all positions computed simultaneously) looks like the model has full sequence access.

The truth: Attention masking enforces causality. At position $t$ , self-attention scores for positions $> t$ are set to $-\infty$ before softmax, zeroing their contribution. Computationally parallel ≠ informationally non-causal.

Fix: Check attention implementation:

mask = torch.triu(torch.ones(seq_len, seq_len), diagonal=1).bool()
scores.masked_fill_(mask, float('-inf'))

Mistake 2: "First token has no loss"

Wrong thinking: "Position 1 has no context, so $\mathcal{L}_1 = 0$ ." Why it feels right: "How can you predict without input?"

The truth: $P_\theta(x_1)$ is the unconditional distribution over first tokens, learned from data. Model learns which tokens commonly start sentences (e.g., "The", "I", capital letters). Loss is $-\log P_\theta(x_1)$ .

Fix: Include position 1 in loss summation. Often, a special `<BOS>` (beginning-of-sequence) token is prepended, so $x_1$ is predicted from `<BOS>`.

Mistake 3: "Minimize loss = generate better text"

Wrong thinking: "Lower loss on validation set always means better generation quality."

Why it feels right: Loss directly measures prediction accuracy.

The truth: Overfitting to training data minimizes loss but produces memorized, non-creative text. Exposure bias (model never sees its own errors during training) causes compounding mistakes at generation time.

Fix: Use perplexity (exponentiated loss) and human evaluation. Sample with temperature/top- $p$ to balance likelihood and diversity: $P'(x_t) \propto P_\theta(x_t \mid x_{<t})^{1/T}, \quad T > 1\text{ for more randomness}$

Why Causal vs. Other Objectives?

Objective	Prediction	Mask	Use Case
Causal LM	Next token	Future positions	Generation (GPT, GPT-2, GPT-3)
Masked LM	Masked tokens	Random positions	Understanding (BERT)
Prefix LM	Next token	Future in suffix	Hybrid (encoder-decoder)

Why causal for generation? At inference, you only have past tokens. Training objective must match test-time reality (no future available).

Why masked LM can't generate? BERT sees bidirectional context, but generation is left-to-right. Mismatch between training and inference.

Recall Explain to a 12-Year-Old

Imagine you're writing a story, one word at a time. You can look back at what you've already written, but you can't peek ahead at what comes next (because you haven't written it yet!).

Causal language modeling is teaching a robot to play this same game. The robot reads "Once upon a", then guesses the next word might be "time". If the real next word is "time", the robot gets a good score. If it guessed "banana", it gets a bad score and learns to do better.

The robot practices on millions of stories, learning patterns like:

"Once upon a ___" → usually "time"
"I feel ___" → often "happy", "sad", "excited"
"The dog ___" → probably a verb like "ran", "barked"

The trick is the robot never cheats by looking ahead. It only sees what it's already written, just like you. That's why it's called "causal"—like cause and effect, the past causes the future, not the other way around!

After enough practice, the robot becomes so good at guessing next words that it can write entire stories on its own, one word at a time.

Connections

Autoregressive Models — Causal LM is the autoregressive paradigm applied to text
Cross-Entropy Loss — The mathematical form of the CLM objective
Transformer Decoder — Architecture implementing causal self-attention
Teacher Forcing — Training technique where true tokens (not predictions) provide context
Perplexity — Evaluation metric: $\exp(\mathcal{L}_{\text{CLM}})$ , lower is better
Masked Language Modeling — Contrasting objective for bidirectional understanding (BERT)
Exposure Bias — Training-inference mismatch in autoregressive models
Sampling Strategies — How to generate from $P_\theta(x_t | x_{<t})$ (greedy, beam search, nucleus)
GPT Family — Models trained with causal LM objective

#flashcards/ai-ml

What is the causal language modeling objective? :: Minimize negative log-likelihood $-\sum_{t=1}^{T} \log P_\theta(x_t \mid x_{<t})$ , predicting each token from only previous tokens (no future access).

Why is it called "causal"? :: Because prediction at position $t$ can only depend on positions before $t$ (cause → effect), preserving time's forward direction.

How does CLM factorize sequence probability?

By the chain rule:

P(\mathbf{x}) = \prod_{t=1}^{T} P(x_t \mid x_{<t})

, decomposing joint probability into conditional next-token predictions.

What enforces causality in transformers?

Attention masking: positions

> t

get scores of

-\infty

before softmax, zeroing their contribution to position

t

's representation.

What is the loss for a single position

t

\mathcal{L}_t = -\log P_\theta(x_t \mid x_{<t})

, the negative log-probability assigned to the true token

x_t

given past context.

Why use negative log-likelihood instead of probability product?

Numerical stability (logs avoid underflow), additivity (easy to sum), and convexity (better optimization properties).

What is the relationship between CLM and cross-entropy?

CLM is cross-entropy between the true next-token distribution (one-hot at correct token) and model's predicted softmax distribution.

How is causal LM used for generation?

Autoregressively sample

x_t \sim P_\theta(\cdot \mid x_{<t})

, then condition on sampled

x_t

to predict

x_{t+1}

, repeating until end-of-sequence.

What is perplexity in context of CLM?

\text{PPL} = \exp(\mathcal{L}_{\text{CLM}})

, the exponentiated average negative log-likelihood; lower perplexity = better predictions.

Why does CLM train on shifted targets?

Input at position

t-1

predicts target at position

t

(input: "The cat sat", targets: "cat sat on"), implementing

P(x_t \mid x_{<t})

What is exposure bias in causal LM?

Training uses true tokens as context (teacher forcing), but generation uses model's own predictions; errors compound since model never practiced recovering from mistakes.

Why can't BERT generate text like GPT?

BERT uses masked LM with bidirectional context, but generation is left-to-right (only past available); training-inference mismatch prevents autoregressive sampling.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho yaar, causal language modeling ka core idea bahut simple hai - model ko sirf yeh sikhana hai ki ek sentence mein "next word" kya aayega, wo bhi sirf pichle words ko dekh ke, future ko bilkul dekhe bina. Jaise aap ek sentence left-to-right padh rahe ho aur har agle word ko guess kar rahe ho - "I..." ke baad "love" aane ki probability zyada hai, "quantum" ki kam. Yeh "causal" isliye kehte hain kyunki position $t$ ki prediction sirf uske pehle wale tokens ( $x_{<t}$ ) pe depend kar sakti hai, matlab cause pehle aata hai aur effect baad mein, time ki direction preserve hoti hai.

Ab maths ki taraf dekho toh yeh chain rule of probability se aata hai. Kisi bhi poore sentence ki probability ko hum chote-chote conditional probabilities ke product mein tod sakte hain - pehle $P(w_1)$ , phir $P(w_2|w_1)$ , aur aise aage. Model ko train karne ke liye hum negative log-likelihood minimize karte hain, jo actually cross-entropy loss ban jaata hai. Log lene ke teen faayde hain: chote probabilities multiply karne se numbers underflow ho jaate hain (log se yeh problem nahi hoti), loss additive ban jaata hai (batch ka total loss bas sum ho jaata hai), aur optimization convex ho jaati hai. Practically, transformer har position pe logits deta hai, softmax se woh probabilities banti hain, aur attention mein "causal masking" lagti hai taaki model cheating karke future tokens na dekh sake.

Yeh matter isliye karta hai kyunki yahi objective aaj ke saare bade text-generation models - jaise GPT family - ke peeche ka fundamental hai. Jab model yeh forward-looking prediction achhe se seekh leta hai, toh wo naya coherent text generate kar sakta hai, ek-ek token karke. Toh agar aap language models ko deeply samajhna chahte ho, yeh objective aapka foundation hai - baaki sab isi ke upar build hota hai.

What Problem Does It Solve?

Derivation from First Principles

Step 1: Start with Maximum Likelihood

Step 2: Apply Chain Rule to Each Sequence

Step 3: Take Negative Log for Optimization

Step 4: Average Over Dataset

Worked Examples

Example 1: Tiny Sequence

Example 2: Generation vs. Training

Why Causal vs. Other Objectives?

Connections

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Tokenization & Language Modeling

Connections