Tokenization & Language Modeling

Q1. (4 marks)

• Tokenization = the process of splitting raw text into discrete units (tokens) — characters, subwords, or words — that are mapped to integer IDs for the model. (2)
• Reason subword preferred: handles rare / out-of-vocabulary words by decomposing them into known subword units, keeping a compact vocabulary while avoiding <UNK> explosion. (2)

Q2. (5 marks)

• Core operation: repeatedly find the most frequent adjacent pair of symbols in the corpus and merge them into a single new symbol, adding it to the vocabulary. (2)
• Count adjacent pairs across the corpus:
  • es: from newest(6) + widest(3) = 9
  • st: from newest(6) + widest(3) = 9
  • lo: 5+2 = 7, ow: 5+2 = 7, we: 2+6 = 8, etc.
• The pair es (or equivalently st) has the highest frequency = 9. First merge = e+s → es, frequency 9. (3) (Accept st with freq 9 if consistently justified.)

Q3. (4 marks)

• BPE merges the pair with the highest raw frequency (count of co-occurrence). (2)
• WordPiece merges the pair that maximizes the likelihood of the training data, i.e. it picks the pair maximizing $\frac{\text{count}(xy)}{\text{count}(x)\,\text{count}(y)}$ (a pointwise-mutual-information-like score), not just raw frequency. (2)

Q4. (4 marks)

• Advantage: larger vocab → shorter token sequences per text, more semantic content per token, fewer subword splits. (2)
• Disadvantage: larger embedding/output matrices → more parameters and memory, rarer tokens seen less often (data sparsity / undertrained embeddings). (2)

Q5. (4 marks)

• Tied embeddings = using the same weight matrix for the input embedding layer and the output (pre-softmax) projection layer. (2)
• Untied: input $V\times d$ + output $V\times d$ = $2Vd$ params. Tied: $Vd$. Saved = $Vd$ parameters. (2)

Q6. (5 marks)

• Objective (negative log-likelihood): $\mathcal{L} = -\sum_{t=1}^{T}\log P(x_t \mid x_1,\dots,x_{t-1};\theta)$ (3)
• "Causal" enforces that the prediction of $x_t$ may depend only on preceding tokens (no access to future tokens) — enforced via a causal/autoregressive mask. (2)

Q7. (4 marks)

• MLM: randomly mask a subset of tokens and predict the masked tokens using bidirectional context (both left and right); loss computed only on masked positions. Causal LM predicts the next token using left context only. (2)
• Typical BERT masking rate = 15% of tokens. (2) (Bonus context: of those, 80% → [MASK], 10% random, 10% unchanged.)

Q8. (4 marks)

• NSP = a binary classification task where the model is given two segments A and B and predicts whether B actually follows A in the original text. (2)
• Labels: IsNext (positive) / NotNext (negative). Positive = B is the real next sentence; negative = B is a random sentence from the corpus (usually 50/50 split). (2)

Q9. (6 marks)

• Perplexity $= \exp\!\left(-\frac{1}{T}\sum_{t=1}^{T}\ln P(x_t)\right) = \left(\prod_t P(x_t)\right)^{-1/T}$. (2)
• Product $= 0.5 \times 0.25 \times 0.5 = 0.0625$. (1)
• $PPL = 0.0625^{-1/3}$. (1)
• $0.0625 = 2^{-4}$, so $PPL = 2^{4/3} = 2^{1.333\ldots} \approx \mathbf{2.52}$. (2)

[
  {"claim":"First BPE merge pair 'es' has frequency 9",
   "code":"es=6+3; result = (es==9)"},
  {"claim":"Tied embeddings save V*d parameters",
   "code":"V,d=sympy.symbols('V d'); untied=2*V*d; tied=V*d; result = sympy.simplify(untied-tied - V*d)==0"},
  {"claim":"Product of per-token probs is 0.0625",
   "code":"p=sympy.Rational(1,2)*sympy.Rational(1,4)*sympy.Rational(1,2); result = (p==sympy.Rational(1,16))"},
  {"claim":"Perplexity of sequence equals 2**(4/3) approx 2.5198",
   "code":"ppl=sympy.Rational(1,16)**(-sympy.Rational(1,3)); result = abs(float(ppl)-2.5198420997897464)<1e-6"}
]