Alignment, Prompting & RAG

Question 1 (14 marks)

(a) (4 marks)

• RLHF trains a reward model $r_\phi$ under the Bradley–Terry model $P(y_w \succ y_l) = \sigma(r(y_w)-r(y_l))$, then optimizes the policy against $r_\phi$ with PPO. (1)
• DPO shows the optimal RLHF policy has a closed form relating reward to policy: $r(x,y) = \beta\log\frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)} + \beta\log Z(x)$. (1)
• Substituting into the Bradley–Terry likelihood, $Z(x)$ cancels (it is the same for $y_w,y_l$ given $x$), giving a loss directly in terms of the policy. (1)
• Hence no separate reward network and no online RL sampling loop are needed — it becomes a supervised classification-style loss on preference pairs. (1)

(b) (6 marks) Compute the two log-ratios:

• $y_w$: $-8.0 - (-9.0) = 1.0$ (1)
• $y_l$: $-6.0 - (-5.0) = -1.0$ (1)
• Difference: $1.0 - (-1.0) = 2.0$; argument $= \beta \cdot 2.0 = 0.1 \times 2.0 = 0.2$ (1)
• Loss $= -\log\sigma(0.2)$. $\sigma(0.2) = 1/(1+e^{-0.2}) = 0.549834$; $\mathcal{L} = -\log(0.549834) = 0.598139$ (2)
• Argument is positive $\Rightarrow$ model already ranks $y_w$ above $y_l$ relative to reference; gradient will reinforce the correct ordering — desired direction. (1)

(c) (4 marks)

• $\hat r(y_w) = 0.1 \times 1.0 = 0.1$ (1)
• $\hat r(y_l) = 0.1 \times (-1.0) = -0.1$ (1)
• Difference $= 0.2 > 0$ (1)
• Positive difference means the implicit reward prefers $y_w$, consistent with the human preference; the policy is aligned on this pair. (1)

Question 2 (14 marks)

(a) (6 marks)

• Choose (iii) hybrid + reranker. (1)
• SKU codes are rare exact tokens: dense embeddings tend to smear/normalize rare literal strings, so dense-only may miss exact code matches (lexical failure). (1.5)
• Natural-language intent (semantics, paraphrase): BM25-only fails on synonyms/paraphrase where no lexical overlap exists (vocabulary mismatch). (1.5)
• Hybrid combines lexical (BM25 catches SKU) + dense (catches intent); reranker (cross-encoder) resolves final ordering with full query-doc attention. (1.5)
• Note reranker adds latency, justified by needing precision — leads into (b). (0.5)

(b) (4 marks)

• Budget for reranking $= 800 - 500 = 300$ ms. (2)
• $k_{\max} = \lfloor 300 / 12 \rfloor = 25$ candidates. (2)

(c) (4 marks)

• Exact cosine over 2M×768 vectors is $O(N)$ per query (~1.5B multiply-adds) → too slow for 800 ms at scale; HNSW gives sub-linear (~log) search. (2)
• Trade-off: ANN is approximate — recall < 100%, so some true nearest neighbors may be missed (recall/latency trade-off tuned by parameters like efSearch). (2)

Question 3 (12 marks)

(a) (5 marks)

• Use structure-aware / semantic chunking on clause boundaries, one clause (or clause group) per chunk, with modest overlap (e.g., 10–20%) to preserve cross-reference context, keeping clause numbers/headers as metadata. (3)
• Naive fixed-256-token chunking can split a clause mid-sentence, orphan a clause number from its body, and break cross-references — so the model sees fragments and invents/mismatches clause numbers. (2)

(b) (6 marks) — one per stage, 2 each:

• Retrieval time: relevance/score thresholding — drop low-similarity chunks so the model isn't fed off-topic context that invites fabrication. (2)
• Generation time: grounding/citation prompting ("answer only from context; cite chunk; say 'not found' if absent") — constrains generation to supplied evidence. (2)
• Evaluation time: faithfulness check / LLM-as-judge or NLI entailment of claims against retrieved sources — flags/filters unsupported statements before returning. (2)

(c) (1 mark)

• More chunks add irrelevant/contradictory context ("lost in the middle", distractors) that the model may blend or misattribute, increasing fabrication. (1)

Question 4 (12 marks)

(a) (5 marks)

• Zero-shot: lowest tokens, lowest accuracy on multi-step problems. (1)
• Few-shot: examples improve format/accuracy, moderate token cost, but may still skip reasoning. (1)
• CoT: highest accuracy on multi-step arithmetic, highest token cost (generates reasoning). (1)
• Mechanism: CoT allocates intermediate compute/tokens, decomposing the problem so each step conditions on prior derived results, reducing single-shot arithmetic errors and giving the model "scratch space." (2)

(b) (5 marks)

• Need $P(\text{at least 3 of 5 correct})$ with $p=0.6$, Binomial(5, 0.6). (1)
• $P(3)=\binom{5}{3}(0.6)^3(0.4)^2 = 10(0.216)(0.16)=0.3456$
• $P(4)=\binom{5}{4}(0.6)^4(0.4)=5(0.1296)(0.4)=0.2592$
• $P(5)=(0.6)^5=0.07776$ (3)
• Sum $=0.3456+0.2592+0.07776 = 0.68256$ (1)

(c) (2 marks)

• ToT preferred when the task needs exploration/backtracking over a branching search space (e.g., puzzles, planning, game-of-24) where intermediate states must be evaluated and pruned — not just independent parallel chains. (2)

Question 5 (8 marks)

(a) (5 marks)

• Phase 1 (SL, critique-revision): model generates a response, critiques it against constitutional principles, then revises; fine-tune on the revised responses. (2)
• Phase 2 (RLAIF): train a preference/reward model from AI-generated preference labels (chosen per principles), then RL-optimize the policy against it. (2)
• Advantage: replaces expensive/slow human labels with scalable AI feedback → cheaper, faster, more consistent harmlessness labeling. (1)

(b) (3 marks)

• Reward hacking: policy exploits flaws/biases in the imperfect reward model, maximizing predicted reward while degrading true quality (e.g., verbosity, sycophancy, formatting tricks). (2)
• Mitigation: KL penalty to reference policy (or early stopping / reward-model refresh / ensembling). (1)

[
  {"claim": "DPO sigmoid argument = 0.2 and loss = 0.598139", "code": "arg = 0.1*((-8.0-(-9.0)) - (-6.0-(-5.0))); import sympy as sp; loss = -sp.log(1/(1+sp.exp(-arg))); result = (abs(arg-0.2)<1e-9) and (abs(float(loss)-0.598139)<1e-4)"},
  {"claim": "Implicit reward difference = 0.2", "code": "rw=0.1*1.0; rl=0.1*(-1.0); result = abs((rw-rl)-0.2)<1e-9"},
  {"claim": "Max reranker candidates k = 25", "code": "k = (800-500)//12; result = k==25"},
  {"claim": "Self-consistency P(>=3 of 5 correct, p=0.6) = 0.68256", "code": "from sympy import binomial, Rational; p=Rational(6,10); q=1-p; total=sum(binomial(5,i)*p**i*q**(5-i) for i in range(3,6)); result = abs(float(total)-0.68256)<1e-6"}
]