Level 1 — RecognitionTokenization & Language Modeling

Tokenization & Language Modeling

⏱ 20 minutes30 marksprintable — key stays hidden on paper

Chapter: 4.2 Tokenization & Language Modeling Level: 1 — Recognition (MCQ, Matching, True/False with justification) Time Limit: 20 minutes Total Marks: 30

Section A — Multiple Choice (1 mark each) [10 marks]

Choose the single best answer.

Q1. Byte-Pair Encoding (BPE) builds its vocabulary by repeatedly: (a) removing the rarest characters (b) merging the most frequent adjacent symbol pair (c) splitting words at whitespace only (d) assigning random subword IDs

Q2. In a causal (autoregressive) language model, the prediction of token $x_t$ may attend to: (a) all tokens including future ones (b) only tokens $x_1 \dots x_{t-1}$ (c) only the current token $x_t$ (d) only masked tokens

Q3. Perplexity of a language model on a sequence is defined as: (a) $\exp$ of the average cross-entropy (in nats) (b) the sum of all token probabilities (c) the number of tokens in the vocabulary (d) the negative log of the largest logit

Q4. WordPiece differs from BPE mainly in that it selects merges to maximize: (a) raw pair frequency (b) the likelihood of the training corpus under the unigram/merged model (c) the vocabulary size (d) whitespace boundaries

Q5. "Tied weights" between the embedding and output projection layers primarily: (a) increase the number of parameters (b) reduce parameters and often improve generalization (c) disable the softmax (d) force a fixed context window

Q6. BERT's Masked Language Modeling (MLM) objective trains the model to predict: (a) the next sentence label (b) randomly masked tokens using bidirectional context (c) only the first token (d) the perplexity value

Q7. Increasing vocabulary size, all else equal, tends to: (a) increase embedding-table parameters but shorten sequences (b) decrease embedding parameters and lengthen sequences (c) have no effect on sequence length (d) eliminate the need for tokenization

Q8. SentencePiece is notable because it: (a) requires pre-tokenized whitespace-split input (b) treats the raw text (including spaces) as a stream, enabling language-agnostic tokenization (c) only works on English (d) is identical to character-level tokenization

Q9. The context window of a Transformer refers to: (a) the number of attention heads (b) the maximum number of tokens the model can process at once (c) the vocabulary size (d) the embedding dimension

Q10. Next Sentence Prediction (NSP) is a binary classification task predicting whether: (a) sentence B logically follows sentence A in the original text (b) a token is masked (c) the sequence exceeds the context window (d) the perplexity is below 1

Section B — Matching (1 mark each) [6 marks]

Q11. Match each term (i–vi) to its correct description (A–F).

Term	Description
(i) BPE	(A) Predicts masked tokens with bidirectional context
(ii) Perplexity	(B) Merges frequent symbol pairs to form subwords
(iii) MLM	(C) Max tokens processable in one forward pass
(iv) Tied weights	(D) Exponentiated average cross-entropy
(v) Context window	(E) Shared input-embedding / output-projection matrix
(vi) Causal LM	(F) Left-to-right next-token prediction

Write six pairs, e.g. (i)–(?).

Section C — True/False WITH Justification (2 marks each: 1 for verdict, 1 for reason) [14 marks]

State True or False and give a one-line justification.

Q12. "A smaller vocabulary always produces shorter tokenized sequences for the same text."

Q13. "Lower perplexity indicates a better-fitting language model."

Q14. "In masked language modeling, the model is allowed to see tokens to the right of the masked position."

Q15. "Perplexity equals $2^{H}$ when the cross-entropy $H$ is measured in bits."

Q16. "Weight tying requires the embedding dimension to equal the model's hidden size so the same matrix can be reused."

Q17. "Byte-level BPE can represent any Unicode string without out-of-vocabulary tokens."

Q18. "The causal language modeling loss is the cross-entropy averaged over predicted next tokens."

Answer keyMark scheme & solutions

Section A (1 mark each)

Q1 — (b). BPE greedily merges the most frequent adjacent symbol pair each iteration, growing subword units. (1)

Q2 — (b). Causal masking blocks future tokens; prediction of $x_t$ conditions only on $x_1\dots x_{t-1}$ . (1)

Q3 — (a). $\text{PPL}=\exp\left(\frac{1}{N}\sum_t -\ln p(x_t\mid x_{<t})\right)$ = exp of mean cross-entropy in nats. (1)

Q4 — (b). WordPiece chooses the merge that maximizes training-data likelihood (score = freq(pair)/(freq(a)·freq(b))), not raw frequency. (1)

Q5 — (b). Sharing the matrix removes a duplicate $V\times d$ table, cutting parameters and regularizing. (1)

Q6 — (b). MLM masks ~15% of tokens and predicts them using both left and right context. (1)

Q7 — (a). Bigger vocab ⇒ larger $V\times d$ embedding table but fewer/longer subwords ⇒ shorter sequences. (1)

Q8 — (b). SentencePiece operates on raw text (spaces encoded, e.g. as ▁), removing language-specific pre-tokenization. (1)

Q9 — (b). Context window = max sequence length (in tokens) per forward pass. (1)

Q10 — (a). NSP predicts whether B is the actual next segment after A (vs. a random one). (1)

Section B (1 mark each)

Q11. (i)–B, (ii)–D, (iii)–A, (iv)–E, (v)–C, (vi)–F. (1 each = 6)

Section C (verdict 1 + reason 1)

Q12 — False. (1) Smaller vocab means words split into more subword units, generally producing longer sequences. (1)

Q13 — True. (1) Lower perplexity = lower average uncertainty per token ⇒ better predictive fit. (1)

Q14 — True. (1) MLM is bidirectional; it uses both left and right context (that is its advantage over causal LM). (1)

Q15 — True. (1) With $H$ in bits, $\text{PPL}=2^{H}$ ; with $H$ in nats, $\text{PPL}=e^{H}$ . (1)

Q16 — True. (1) The shared matrix has shape $V\times d$ for input and $d\times V$ (transpose) for output, so both must use the same $d$ . (1)

Q17 — True. (1) Byte-level BPE starts from the 256 possible bytes, so every UTF-8 string is representable — no OOV. (1)

Q18 — True. (1) Causal LM loss $=\frac{1}{N}\sum_t -\ln p(x_t\mid x_{<t})$ , the mean next-token cross-entropy. (1)

Worked numeric illustration (supports Q3, Q13, Q15)

Suppose a model assigns probabilities $[0.5, 0.25, 0.5, 0.25]$ to 4 successive correct tokens. Cross-entropy in nats: $H=-\frac14(\ln0.5+\ln0.25+\ln0.5+\ln0.25)=\ln2\cdot\frac{6}{4}=1.5\ln2\approx1.0397$ . Perplexity: $e^{H}=2^{1.5}\approx2.828$ . In bits $H=1.5$ , and $2^{1.5}=2.828$ matches — confirming $e^{H_{nats}}=2^{H_{bits}}$ .

[
  {"claim":"Perplexity = exp(mean cross-entropy nats) = 2^(mean CE bits) for probs [0.5,0.25,0.5,0.25]",
   "code":"import sympy as sp\np=[sp.Rational(1,2),sp.Rational(1,4),sp.Rational(1,2),sp.Rational(1,4)]\nH_nats=-sum(sp.log(x) for x in p)/4\nH_bits=-sum(sp.log(x,2) for x in p)/4\nppl_nats=sp.exp(H_nats)\nppl_bits=2**H_bits\nresult=sp.simplify(ppl_nats-ppl_bits)==0 and sp.simplify(ppl_nats-2*sp.sqrt(2))==0"},
  {"claim":"Uniform model over V tokens has perplexity exactly V",
   "code":"import sympy as sp\nV=sp.Symbol('V',positive=True)\np=1/V\nH=-sp.log(p)\nppl=sp.exp(H)\nresult=sp.simplify(ppl-V)==0"},
  {"claim":"Weight tying removes one V*d embedding table (param saving = V*d)",
   "code":"import sympy as sp\nV,d=sp.symbols('V d',positive=True)\nuntied=2*V*d\ntied=V*d\nresult=sp.simplify(untied-tied-V*d)==0"}
]