Alignment, Prompting & RAG

Question 1 (22 marks)

(a) Optimal policy derivation (8)

Write the objective per prompt $x$ as a functional of $\pi(\cdot|x)$: $J(\pi)=\sum_y \pi(y|x)\,r(x,y) - \beta\sum_y \pi(y|x)\log\frac{\pi(y|x)}{\pi_{\text{ref}}(y|x)}.$ (1 mark for correct setup, KL expanded.)

Maximize subject to $\sum_y\pi(y|x)=1$. Lagrangian: $\mathcal{L}=J(\pi)+\lambda\Big(1-\sum_y\pi(y|x)\Big).$ (1) Differentiate w.r.t. $\pi(y|x)$: $r(x,y)-\beta\Big(\log\frac{\pi(y|x)}{\pi_{\text{ref}}(y|x)}+1\Big)-\lambda=0.$ (2 — the "+1" from $\frac{d}{d\pi}(\pi\log\pi)$.)

Solve: $\log\frac{\pi(y|x)}{\pi_{\text{ref}}(y|x)}=\frac{1}{\beta}r(x,y)-1-\frac{\lambda}{\beta}$ $\Rightarrow \pi^*(y|x)=\pi_{\text{ref}}(y|x)\exp\!\big(\tfrac{1}{\beta}r(x,y)\big)\cdot e^{-1-\lambda/\beta}.$ (2) The constant absorbs into normalization; enforcing $\sum_y\pi^*=1$ gives $\pi^*(y|x)=\frac{1}{Z(x)}\pi_{\text{ref}}(y|x)\exp\!\big(\tfrac1\beta r(x,y)\big),\quad Z(x)=\sum_y \pi_{\text{ref}}(y|x)\exp\!\big(\tfrac1\beta r(x,y)\big).$ (1 for $Z$.) $Z(x)$ sums over the entire (exponentially large / infinite) space of output sequences $y$, so it is intractable to compute or sample from exactly. (1)

(b) Reward inversion + $Z$ cancellation (8)

Invert: $r(x,y)=\beta\log\frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)}+\beta\log Z(x). \quad (2)$

Under Bradley–Terry, only the difference of rewards enters: $r(x,y_w)-r(x,y_l)=\beta\log\frac{\pi^*(y_w|x)}{\pi_{\text{ref}}(y_w|x)}-\beta\log\frac{\pi^*(y_l|x)}{\pi_{\text{ref}}(y_l|x)}+\underbrace{\beta\log Z(x)-\beta\log Z(x)}_{=0}. \quad (3)$ The $\beta\log Z(x)$ terms cancel because $Z(x)$ depends only on $x$, not on $y$. (1)

Substituting into $-\log\sigma(\cdot)$ (negative log-likelihood of the preference), and replacing $\pi^*$ with the trainable $\pi_\theta$: $\mathcal{L}_{\text{DPO}}=-\log\sigma\!\Big(\beta\log\tfrac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)}-\beta\log\tfrac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\Big). \quad (2)$

(c) Gradient and interpretation (6)

Let $u=\beta\big(\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)}-\log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\big)$, i.e. the difference of implicit rewards $\hat r_\theta(y_w)-\hat r_\theta(y_l)$. Since $\frac{d}{du}(-\log\sigma(u))=-(1-\sigma(u))=-\sigma(-u)$: $\nabla_\theta\mathcal{L}_{\text{DPO}}=-\beta\,\sigma\big(\hat r_\theta(y_l)-\hat r_\theta(y_w)\big)\Big[\nabla_\theta\log\pi_\theta(y_w|x)-\nabla_\theta\log\pi_\theta(y_l|x)\Big]. \quad (4)$

Interpretation (2): The scalar $\sigma(\hat r_\theta(y_l)-\hat r_\theta(y_w))$ is large (near 1) exactly when the model currently ranks the dispreferred response higher than the preferred one — i.e. the implicit reward is "wrong." So mis-ranked pairs get large gradient weight; correctly-ranked confident pairs get near-zero weight. This is automatic hard-example weighting, no separate reward model needed.

---

Question 2 (18 marks)

(a) Numeric surrogate values (8)

Positive advantage $\hat A_t=+3$, $\epsilon=0.2$ so clip range $[0.8,1.2]$: $L^{\text{CLIP}}=\min(r\cdot3,\ \mathrm{clip}(r,0.8,1.2)\cdot3)$.

• $r=0.5$: $\min(1.5,\ 0.8\cdot3=2.4)=\mathbf{1.5}$ (unclipped term smaller). (1)
• $r=1.0$: $\min(3,3)=\mathbf{3.0}$. (1)
• $r=1.5$: $\min(4.5,\ 1.2\cdot3=3.6)=\mathbf{3.6}$ (clipped). (1)

Shape: rises linearly, then flat ceiling of $3.6$ for $r\ge1.2$; below $r=0.8$ it stays the unclipped line (since min picks smaller). So the objective is capped above for large $r$. (1)

Negative advantage $\hat A_t=-3$: $L=\min(-3r,\ \mathrm{clip}(r,0.8,1.2)\cdot(-3))$.

• $r=0.5$: $\min(-1.5,\ 0.8\cdot(-3)=-2.4)=\mathbf{-2.4}$ (clipped). (1)
• $r=1.0$: $\min(-3,-3)=\mathbf{-3.0}$. (1)
• $r=1.5$: $\min(-4.5,\ 1.2\cdot(-3)=-3.6)=\mathbf{-4.5}$ (unclipped). (1)

Shape: floored below at $-2.4$ for $r\le0.8$; falls linearly for $r>1.2$. (1)

(b) Pessimistic lower bound & one-sidedness (6)

Because $L^{\text{CLIP}}=\min(\text{unclipped},\ \text{clipped})$, it never exceeds the unclipped surrogate — it is a lower bound on the true (unclipped) objective, hence "pessimistic." (2)

Directionality: (4)

• For $\hat A_t>0$ (good action): we want to increase $\pi_\theta(a_t)$, i.e. increase $r_t$. Clipping caps the gain once $r_t>1+\epsilon$, so it removes incentive to push $r_t$ too high. But decreasing $r_t$ (below 1) is not clamped — the gradient still flows. So the constraint is one-sided: it limits over-eager increases, not decreases.
• For $\hat A_t<0$ (bad action): symmetric — clipping limits pushing $r_t$ below $1-\epsilon$, but does not constrain increasing $r_t$. Net effect: only the update direction that would exploit a large ratio to gain surrogate is clamped; the corrective direction remains free.

(c) Reward hacking / over-optimization (4)

Failure mode: reward model over-optimization (reward hacking / Goodhart's law) — the policy exploits errors in $r_\phi$, so proxy reward rises while true (human) quality falls. (2)

Two mitigations tied to the objective (2, one each):

1. Increase $\beta$ (KL penalty) to keep $\pi_\theta$ close to $\pi_{\text{ref}}$, limiting how far it can drift to exploit $r_\phi$.
2. Early stopping / KL budget (stop when $\mathrm{KL}(\pi_\theta\|\pi_{\text{ref}})$ exceeds a threshold), or retrain/ensemble the reward model on newly generated on-policy samples to close the exploited gap.

---

Question 3 (20 marks)

(a) Cosine similarities + equivalence (8)

$q=(0.6,0.8)$, $\|q\|=1$.

• $c_1=(3,4)$, $\|c_1\|=5$, $\hat c_1=(0.6,0.8)$. $\cos=0.6\cdot0.6+0.8\cdot0.8=0.36+0.64=\mathbf{1.0}$. (2)
• $c_2=(1,0)$, $\|c_2\|=1$. $\cos=0.6\cdot1+0.8\cdot0=\mathbf{0.6}$. (1)
• $c_3=(-0.8,0.6)$, $\|c_3\|=1$. $\cos=0.6(-0.8)+0.8(0.6)=-0.48+0.48=\mathbf{0}$. (1)

Top-1: $c_1$ (similarity 1.0 — same direction as $q$). (1)

Equivalence (3): For unit vectors $\|a\|=\|b\|=1$, $\|a-b\|^2=\|a\|^2+\|b\|^2-2a\!\cdot\!b=2-2\cos(a,b).$ So squared Euclidean distance is a strictly decreasing affine function of cosine similarity; ascending distance order = descending cosine order. Ranking is identical.

(b) Complexity ratio (4)

Exact: $O(Nd)=10^6\cdot768=7.68\times10^8$ ops. (1) Approx: $O(d\log_2 N)=768\cdot\log_2(10^6)\approx768\cdot19.93\approx1.53\times10^4$ ops. (1) Ratio $\approx \frac{7.68\times10^8}{1.53\times10^4}\approx \mathbf{5.0\times10^4}$ (≈50,000× fewer ops). (1) Recall < 100% because ANN graph traversal (HNSW) is heuristic/greedy and may miss true neighbours near cluster boundaries — it trades accuracy for the huge speedup. (1)

(c) Hybrid pipe