4.2.6 · D5Tokenization & Language Modeling
Question bank — Causal language modeling objective
Before we start, one word we lean on: causal means prediction at position may use only tokens at positions — the past, never the future. Keep that picture in mind (a reader with a hand covering everything to the right of the current word).
True or false — justify
TRUE/FALSE: The causal LM loss ignores the very first token because it has no context.
False — is the unconditional distribution over openers, learned from data (often via a prepended
<BOS> token). It is included in the sum.TRUE/FALSE: Because all positions are computed in one parallel forward pass, the model effectively sees the whole sentence when predicting each token.
False — parallel computation is not the same as information access. The Transformer Decoder attention mask sets scores for future positions to , so position never reads .
TRUE/FALSE: Lower validation loss always means better generated text.
False — loss measures next-token prediction under Teacher Forcing, not free-running quality. Exposure Bias and sampling choices (Sampling Strategies) can make a low-loss model produce degenerate text.
TRUE/FALSE: The chain-rule factorization is only valid for left-to-right ordering.
False — the chain rule holds for any ordering of the variables; left-to-right is a choice that matches how text is generated, not a mathematical requirement.
TRUE/FALSE: Cross-entropy loss and negative log-likelihood are different objectives here.
False — because the true next-token distribution is one-hot, cross-entropy collapses to exactly ; they are the same number.
TRUE/FALSE: A perfect model achieves zero loss on any real text corpus.
False — real language is genuinely uncertain (many valid continuations). The floor is the data's entropy, which is positive; zero loss would require language to be deterministic.
TRUE/FALSE: Masked Language Modeling (BERT-style) and causal LM optimize the same objective.
False — MLM predicts randomly masked tokens using both left and right context; causal LM predicts the next token using only left context. Only causal LM factorizes cleanly for generation.
TRUE/FALSE: Averaging the loss over the dataset changes which parameters are optimal.
False — dividing by a constant rescales the loss but not its argmin; it only stabilizes the gradient magnitude across batch sizes.
Spot the error
ERROR: "We compute logits, then loss cross_entropy(logits, input_ids) — done."
Missing the shift. Logits at position must be scored against the label at position ; you compare
logits[:, :-1] with input_ids[:, 1:], else the model is asked to predict the token it already saw.ERROR: "Softmax over the vocabulary is optional; we can read the probability straight from the logit."
A raw logit is unnormalized and can be negative — it is not a probability. You must exponentiate and divide by so the values are non-negative and sum to .
ERROR: "To enforce causality, we mask future tokens after the softmax by zeroing their weights."
Too late — masking after softmax breaks normalization (weights no longer sum to 1). Add to future scores before softmax so they receive exactly zero probability and the rest renormalize correctly.
ERROR: "Padding tokens are just more training targets, so we include them in the loss."
Padding carries no linguistic signal; scoring it teaches the model nothing and dilutes the gradient. Use
ignore_index=pad_token_id so padded positions are excluded from the loss.ERROR: "We maximize the product of per-token probabilities directly for numerical stability."
Products of many small probabilities underflow to zero in floating point. That is precisely why we take logs and sum — turning a fragile product into a stable additive loss.
ERROR: "Training on teacher-forced targets guarantees the model handles its own generated prefixes well."
This is the Exposure Bias trap — at training time the model always sees the gold prefix, never its own mistakes, so errors can compound at generation time in ways the loss never penalized.
Why questions
WHY do we take the negative log rather than just the log?
We want to minimize a loss, but likelihood should be maximized; negating flips maximization into minimization and keeps the loss non-negative (since for ).
WHY does the same trained conditional serve both training and generation?
Training fits this conditional at every position simultaneously; generation just applies it repeatedly, feeding each sampled token back in — the Autoregressive Models loop reuses the exact distribution learned.
WHY is a randomly initialized GPT-2's loss about ?
With no training, every one of the tokens is roughly equally likely, so and — the uniform-distribution baseline.
WHY does causal masking make the model "predict the future" without cheating?
The mask removes the future from the input to each prediction, so any correct guess must come from learned patterns in the past, not from copying the answer that sits to the right.
WHY is Perplexity often reported instead of raw loss?
Perplexity converts the log-scale loss into an intuitive "effective number of equally-likely choices" — a perplexity of 20 means the model is as uncertain as a fair 20-sided die per token.
WHY can two models with identical validation loss still differ in output style?
Loss only pins down the average next-token distribution; how you sample from it (greedy vs. temperature vs. top-, see Sampling Strategies) shapes creativity, repetition, and coherence independently of the loss.
Edge cases
EDGE: What is the loss contribution of a single token the model predicts with probability exactly ?
Zero — . The token was perfectly anticipated, so it adds nothing to the loss.
EDGE: What happens to the loss as the model's probability for the true token approaches ?
It diverges to — as . One confidently-wrong prediction can dominate the whole loss, which is why numerical clipping/label smoothing is sometimes used.
EDGE: A sequence of length (a single token) — is there any loss to compute?
Yes, exactly one term , the unconditional opener probability. There is no conditional term because there is nothing before it.
EDGE: The corpus has one sequence repeated a million times and loss hits near — is the model good?
No — this is pure memorization/overfitting. Near-zero training loss with no diversity signals the model captured this exact string, not the language, and it will generalize poorly.
EDGE: The vocabulary contains a token that never appears in training. What probability does the model learn for it?
Some small but non-zero probability from softmax (which can never output an exact 0), spread by generalization — but it is never rewarded for that token, so its estimate stays essentially arbitrary.
Recall One-line summary
Causal LM minimizes summed over left-context-only predictions; every trap here is either a causality leak, a shift/normalization slip, or confusing loss with generation quality.