Alignment, Prompting & RAG

Q1. (4 marks)

• Definition (1): RLHF is a technique to align an LLM with human preferences by using human-labeled preference data to train a reward model, then optimizing the LLM with reinforcement learning against that reward.
• Three stages (3, 1 each):
  1. Supervised fine-tuning (SFT) on demonstration data.
  2. Reward model training on human preference comparisons.
  3. RL optimization (e.g., PPO) of the policy against the reward model.

Q2. (5 marks) (a) (2) Loss over preference dataset $D$: $\mathcal{L}(r) = -\mathbb{E}_{(x,y_w,y_l)\sim D}\big[\log \sigma\big(r(x,y_w)-r(x,y_l)\big)\big]$ (b) (3) Difference $= 2.0 - 0.5 = 1.5$ (1). $P = \sigma(1.5) = \frac{1}{1+e^{-1.5}} = \frac{1}{1+0.22313} = 0.81757 \approx \mathbf{0.818}$ (2 marks: setup + value).

Q3. (6 marks)

• Objective (3): $L^{CLIP}(\theta) = \mathbb{E}_t\Big[\min\big(r_t(\theta)\hat{A}_t,\; \text{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t\big)\Big]$
• Ratio (2): $r_t(\theta) = \dfrac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{old}}(a_t\mid s_t)}$ — ratio of new to old policy probabilities.
• Why clip (1): Clipping prevents excessively large policy updates that would destabilize training by removing the incentive to move the ratio far from 1.

Q4. (6 marks) (a) (2): DPO removes the need to train a separate reward model and to run unstable RL; it directly optimizes the policy on preference pairs with a simple classification-style loss (more stable, simpler, cheaper). (Any one valid advantage.) (b) (4):

• $\pi_{\text{ref}}$ = the reference policy, typically the frozen SFT model, used as a baseline to keep the trained policy from drifting too far (2).
• $\beta$ = temperature/regularization coefficient controlling the strength of the KL constraint to $\pi_{\text{ref}}$; larger $\beta$ = tighter constraint / smaller updates (2).

Q5. (5 marks)

• Zero-shot (1): Task specified by instruction only, no examples given.
• Few-shot (1): A few input–output examples are included in the prompt to demonstrate the task.
• Chain-of-thought (1): Prompt elicits intermediate reasoning steps before the final answer.
• Example cue (2): e.g., "Let's think step by step."

Q6. (6 marks) (a) (3): Instead of taking one greedy chain, self-consistency samples multiple diverse reasoning paths (with temperature sampling) and marginalizes over reasoning by taking a majority vote on the final answers, improving robustness since correct answers are reached via multiple paths. (b) (3): Count: 42 appears 3 times, 17 once, 30 once. Selects 42 by majority vote / plurality rule (3).

Q7. (6 marks) Sequence (any reasonable ordering; ~1.5 each for 4 named):

1. Query embedding — user query encoded into a vector.
2. Retrieval — vector database / index searched for top-$k$ relevant chunks (from pre-indexed, chunked, embedded documents).
3. (Optional) Reranking — retrieved candidates re-scored.
4. Augmentation — retrieved context inserted into the prompt.
5. Generation — LLM produces the grounded answer. Full marks for ≥4 correct components in logical order.

Q8. (6 marks) (a) (4): Dot product $= (1)(0)+(0)(1)+(1)(1) = 1$ (1). $\|\mathbf{a}\| = \sqrt{2},\ \|\mathbf{b}\| = \sqrt{2}$ (1). $\cos\theta = \frac{1}{\sqrt{2}\cdot\sqrt{2}} = \frac{1}{2} = \mathbf{0.5}$ (2). (b) (2): Cosine ignores vector magnitude and measures only directional similarity, which is more meaningful for embeddings where length varies with e.g. text length/frequency and only semantic direction matters.

Q9. (3 marks) Any three (1 each): RAG / grounding in retrieved sources; requiring citations / attribution; lower decoding temperature; self-consistency or self-verification; fine-tuning on factual/honesty data; abstention ("I don't know") training; fact-checking / verifier models.

Q10. (3 marks)

• Definition (2): Using a strong LLM to score, rank, or judge the quality of another model's outputs against a rubric or reference, in place of human evaluators.
• Limitation (1): e.g., position/verbosity/self-preference bias, inconsistency, or cost; may not correlate with true human judgment.

[
  {"claim":"sigma(1.5) rounds to 0.818","code":"val=1/(1+exp(Rational(-3,2))); result=(round(float(val),3)==0.818)"},
  {"claim":"Reward difference is 1.5","code":"result=(Rational(20,10)-Rational(5,10)==Rational(3,2))"},
  {"claim":"Cosine similarity of (1,0,1) and (0,1,1) is 0.5","code":"a=Matrix([1,0,1]); b=Matrix([0,1,1]); cs=(a.dot(b))/(a.norm()*b.norm()); result=(simplify(cs-Rational(1,2))==0)"},
  {"claim":"Majority vote of [42,42,17,42,30] is 42","code":"from collections import Counter; c=Counter([42,42,17,42,30]); result=(c.most_common(1)[0][0]==42)"}
]