Alignment, Prompting & RAG

Q1 — DPO Derivation (12)

(a) (4) Rewrite objective per-prompt as minimizing $\mathbb{E}_{y\sim\pi}\Big[\log\frac{\pi(y|x)}{\pi_{\mathrm{ref}}(y|x)} - \tfrac{1}{\beta}r(x,y)\Big].$ This equals $\mathrm{KL}\big(\pi \,\|\, \tfrac{1}{Z}\pi_{\mathrm{ref}}e^{r/\beta}\big) - \log Z(x)$ with $Z(x)=\sum_y \pi_{\mathrm{ref}}(y|x)e^{r(x,y)/\beta}$. (2 marks) KL is minimized (=0) when $\pi=\pi^*$, giving $\pi^*(y|x)=\tfrac{1}{Z(x)}\pi_{\mathrm{ref}}(y|x)e^{r(x,y)/\beta}.$ (2 marks)

(b) (2) Take logs and rearrange: $r(x,y)=\beta\log\frac{\pi^*(y|x)}{\pi_{\mathrm{ref}}(y|x)}+\beta\log Z(x).$

(c) (3) In BT, only the difference matters: $r(x,y_w)-r(x,y_l)=\beta\log\frac{\pi^*(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)}-\beta\log\frac{\pi^*(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}.$ The $\beta\log Z(x)$ term is identical in both and cancels (1 mark for identifying cancellation; 2 for correct difference).

(d) (3) Replacing $\pi^*$ with the trainable $\pi_\theta$: $\mathcal{L}_{\text{DPO}}=-\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\Big[\log\sigma\Big(\beta\log\tfrac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)}-\beta\log\tfrac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}\Big)\Big].$

Q2 — PPO Clip (10)

(a) (5) $L^{\text{CLIP}}(\theta)=\mathbb{E}_t\big[\min(r_t A_t,\ \mathrm{clip}(r_t,1-\epsilon,1+\epsilon)A_t)\big]$, where $r_t=\frac{\pi_\theta(a_t|s_t)}{\pi_{\text{old}}(a_t|s_t)}$. (3) Why: the clip caps how far the new policy can move from the old one in a single update; taking the min makes the bound pessimistic so the objective never rewards moving outside the trust region — preventing large, destabilizing policy jumps. (2)

(b) (2) $r_tA_t=1.4\times2=2.8$; clipped ratio $=1.2$, $1.2\times2=2.4$; $\min(2.8,2.4)=\mathbf{2.4}$.

(c) (3) $r_tA_t=0.5\times(-3)=-1.5$; clipped ratio $=\max(0.5,0.8)=0.8$, $0.8\times(-3)=-2.4$; $\min(-1.5,-2.4)=\mathbf{-2.4}$. (2) Case (b): clip is active, ratio was above $1+\epsilon$ with positive $A$ ⇒ gradient zero (no further push). Case (c): the $-2.4$ (unclipped, larger magnitude) is selected ⇒ gradient nonzero, pushing probability down. (1)

Q3 — Reward Model (8)

(a) (3) $\mathcal{L}(\phi)=-\mathbb{E}_{(y_w,y_l)}\big[\log\sigma\big(r_\phi(x,y_w)-r_\phi(x,y_l)\big)\big].$

(b) (3) Loss per pair $=-\log\sigma(m)$:

• $m=2$: $-\log\sigma(2)=-\log(0.8808)=0.1269$
• $m=0$: $-\log(0.5)=0.6931$
• $m=-1$: $-\log\sigma(-1)=-\log(0.2689)=1.3133$

Mean $=(0.1269+0.6931+1.3133)/3=\mathbf{0.7111}$.

(c) (2) Init from SFT: shares language understanding, so preference signal only needs to fit a ranking head — cheaper and more stable. Scalar head: BT model needs a single real-valued score per response to compute margins.

Q4 — Self-Consistency (10)

(a) (4) Majority correct $=\sum_{k=3}^{5}\binom{5}{k}p^k(1-p)^{5-k}$, $p=0.55$.

(b) (3)

• $k=3$: $10\cdot0.55^3\cdot0.45^2=10\cdot0.166375\cdot0.2025=0.336909$
• $k=4$: $5\cdot0.55^4\cdot0.45=5\cdot0.09150625\cdot0.45=0.205889$
• $k=5$: $0.55^5=0.050328$

Sum $=\mathbf{0.5931}$ (≈0.593), higher than single-sample $0.55$. ✓

(c) (3) The assumption of independence of samples fails: chains sampled from the same model are correlated (shared biases/failure modes), and wrong answers can cluster on a single plausible-but-wrong value that beats the correct answer under majority vote — collapsing the gain or reversing it.

Q5 — RAG Pipeline (12)

(a) (8) Reference solution:

def rag_query(query, store, llm, embed, bm25, k=20, alpha=0.6):
    q_vec = embed(query)                              # (a) embed
    # (b) dense retrieval by cosine
    cand = store.search(q_vec, top_k=k, metric="cosine")
    # (c) hybrid rerank
    for c in cand:
        s_dense = cosine(q_vec, c.vec)
        s_bm25  = bm25.score(query, c.text)
        c.score = alpha*s_dense + (1-alpha)*normalize(s_bm25)
    top = sorted(cand, key=lambda c: c.score, reverse=True)[:5]
    # (d) grounded prompt
    ctx = "\n\n".join(f"[{i}] {c.text}" for i, c in enumerate(top))
    prompt = (f"Answer ONLY from context. Cite [i]. "
              f"If not present, say 'not in context'.\n\n"
              f"Context:\n{ctx}\n\nQ: {query}\nA:")
    return llm(prompt)                                # (e) call LLM

Marks: embed(1), retrieval(2), hybrid rerank(2), grounded/cited prompt(2), LLM call(1).

(b) (4) — 1 mark each:

• Chunk size trade-off: large chunks preserve context but dilute embeddings & waste context window; small chunks are precise but fragment meaning across boundaries.
• Overlap trade-off: sliding-window overlap avoids splitting an answer across chunks but inflates index size and causes duplicate retrievals.
• Hallucination mitigation: grounding instruction + "say 'not in context'" abstention reduces fabrication.
• Hallucination mitigation: inline citation [i] forces attribution, enabling faithfulness checks / self-verification.

Q6 — Cosine (8)

(a) (5) $q=[1,2,2]$, $\|q\|=3$.

• $c_1=[2,0,1]$: dot $=2+0+2=4$, $\|c_1\|=\sqrt5$, cos $=4/(3\sqrt5)=0.5963$.
• $c_2=[0,3,4]$: dot $=0+6+8=14$, $\|c_2\|=5$, cos $=14/15=0.9333$.

Top-1 retrieved: $c_2$. (dot each 1, norms 1, values 2, decision 1)

(b) (3) After L2 normalization $\|c\|=1$, so $q\cdot c=\frac{q\cdot c}{\|q\|\|c\|}$·(const) — inner product equals cosine up to the fixed $\|q\|$ which is monotone-invariant for ranking. ANN indexes (IVF/HNSW) use raw inner-product/L2 distance kernels that are cheaper than dividing by norms per comparison, so pre-normalizing lets the fast IP search return exact cosine order.

[
{"claim":"Q2b clipped surrogate = 2.4","code":"rt=Rational(14,10);At=2;eps=Rational(2,10);val=Min(rt*At, Max(1-eps,Min(rt,1+eps))*At);result=(val==Rational(24,10))"},
{"claim":"Q2c clipped surrogate = -2.4","code":"rt=Rational(5,10);At=-3;eps=Rational(2,10);val=Min(rt*At, Max(1-eps,Min(rt,1+eps))*At);result=(val==Rational(-24,10))"},
{"claim":"Q3b mean reward-model loss approx 0.7111","code":"import sympy as sp;f=lambda m:-sp.log(1/(1+sp.exp(-m)));mean=(f(2)+f(0)+f(-1))/3;result=abs(float(mean)-0.7111)<1e-3"},
{"claim":"Q4b self-consistency prob approx 0.5931","code":"p=Rational(55,100);from sympy import binomial;s=sum(binomial(5,k)*p**k*(1-p)**(5-k) for k in range(3,6));result=abs(float(s)-0.5931)<1e-3"},
{"claim":"Q6a cosine c2 > c1 so c2 top1","code":"import sympy as sp;q=sp.Matrix([1,2,2]);c1=sp.Matrix([2,0,1]);c2=sp.Matrix([0,3,4]);cos1=q.dot(c1)/(q.norm()*c1.norm());cos2=q.dot(c2)/(q.norm()*c2.norm());result=(cos2>cos1)"}
]