Exercises — Reinforcement Learning from Human Feedback (RLHF)

1. Supervised Fine-Tuning (SFT) — ordinary cross-entropy on (prompt, ideal answer) pairs.
2. Reward Model — trained with the Bradley-Terry Model pairwise loss.
3. RL optimization — Proximal Policy Optimization (PPO) with a KL Divergence penalty.

Mnemonic: "SFT Ranks, PPO Pulls (on a KL leash)."

$y_w$ is the winner (the response the human preferred); $y_l$ is the loser. The output $\sigma (\cdot )$ is the model's predicted probability that the human prefers the winner — a number in $(0,1)$. We take $-\log$ of it so that a confident correct prediction (near $1$) gives near-zero loss.

False. Bradley–Terry only fixes reward differences. Adding the same constant to every reward leaves the loss unchanged, so the zero-point is arbitrary. Only comparisons carry meaning.

• Gap: $\Delta =1.2-(-0.3)=1.5$. Why the gap? The RM only sees the difference of rewards.
• Probability: $\sigma (1.5)=\frac{1}{1+e^{-1.5}}\approx 0.8176$.
• Loss: $-\log (0.8176)\approx 0.2014$.

Low loss — the model already ranks the winner above the loser, so only a gentle nudge is needed.

• Gap: $\Delta =-0.3-1.2=-1.5$.
• Probability: $\sigma (-1.5)\approx 0.1824$.
• Loss: $-\log (0.1824)\approx 1.7014$.

The loss jumped from $\approx 0.20$ to $\approx 1.70$ — an 8.4× larger penalty. A wrong ranking is punished hard, exactly what we want.

• Log-ratio: $\log (0.20/0.05)=\log 4\approx 1.3863$.
• Penalty: $\beta \times \text{log-ratio}=0.2\times 1.3863\approx 0.2773$.

The policy is $4\times$ more confident than the trusted model here, so the reward gain from this token must beat $\approx 0.277$ or the leash pulls it back.

Replace $r_{\theta }(x,y)\to r_{\theta }(x,y)+c$ for all $y$. Inside the loss we only ever use the gap: $(r_{\theta }(x,y_w)+c)-(r_{\theta }(x,y_l)+c)=r_{\theta }(x,y_w)-r_{\theta }(x,y_l).$ The $c$ cancels, so $\sigma (\cdot )$ and the loss are identical. Conclusion: the RM's absolute zero-point is unidentified — only differences are learnable. This is why we never treat a raw reward number as an "absolute grade".

The full objective is ${\max }_{\varphi }{\mathbb{E}}_{y\sim {\pi }_{\varphi }}[r_{\theta }(x,y)-\beta \log \frac{{\pi }_{\varphi }}{{\pi }^{\text{SFT}}}].$ With $\beta =0$ the policy maximizes $r_{\theta }$ alone. But $r_{\theta }$ is a proxy, accurate only near the data it saw. The optimizer will happily wander into weird off-distribution outputs where $r_{\theta }$ is mistakenly high — gibberish that fools the RM. The KL term charges $\beta$ per unit of drift from the trusted ${\pi }^{\text{SFT}}$, so straying far costs reward and is only worth it if the reward gain is real. This is KL Divergence acting as a leash.

Divide numerator and denominator by $e^{r_w}$: $P=\frac{1}{1+e^{r_l-r_w}}=\frac{1}{1+e^{-(r_w-r_l)}}=\sigma (r_w-r_l).$ What this reveals: the model depends only on the difference $r_w-r_l$ — precisely the information a pairwise comparison provides. Everything shared cancels.

• Ratio: ${\rho }_t=0.26/0.20=1.30$.
• Clip range: $[1-0.2,1+0.2]=[0.8,1.2]$, so clipped ${\rho }_t=1.20$.
• Two candidates: ${\rho }_t{A}_t=1.30\times 3.0=3.90$; clipped $=1.20\times 3.0=3.60$.
• PPO term: $\min (3.90,3.60)=3.60$.

Because $A_t>0$ and ${\rho }_t$ overshot $1.2$, Proximal Policy Optimization (PPO) caps the gain at $3.60$ — the model gets no extra reward for pushing past the clip, killing the incentive to take a reckless step.

• Candidates: ${\rho }_t{A}_t=1.30\times (-3.0)=-3.90$; clipped $=1.20\times (-3.0)=-3.60$.
• PPO term: $\min (-3.90,-3.60)=-3.90$.

With a negative advantage the $\min$ picks the more negative (unclipped) value, so the full $-3.90$ penalty applies. This is deliberate: PPO always keeps the more pessimistic estimate, so a genuinely bad action is never let off the hook by clipping.

• Log-ratio: $\log (0.40/0.10)=\log 4\approx 1.3863$.
• Penalty: $0.2\times 1.3863\approx 0.2773$.
• Adjusted reward: $r^{\prime }=0.9-0.2773\approx 0.6227>0$.

Still positive, so the reward gain ($0.9$) clearly beats the leash cost ($0.277$) — the update is justified. Had $r_{\theta }$ been, say, $0.2$, we'd get $r^{\prime }\approx -0.077<0$ and the leash would win.

• Diagnosis: Reward Hacking and Goodhart's Law. The measured proxy (RM reward) improves while the true target (human judgment) degrades — the classic Goodhart split.
• Knob: $\beta$ (the KL coefficient) is too small, so the leash to ${\pi }^{\text{SFT}}$ is too loose and the policy drifts into off-distribution outputs the RM mis-scores.
• Fixes: raise $\beta$; refresh the RM with fresh comparisons on the current policy's outputs (the region it now visits); add early stopping on a held-out human eval. Optionally switch to Direct Preference Optimization (DPO) which folds the KL constraint into a single supervised loss, or use Constitutional AI / RLAIF to scale comparison labels.

• Method: Direct Preference Optimization (DPO).
• How: DPO algebraically shows the optimal RLHF policy is a closed-form function of the reward, so it re-parametrizes the reward through the policy itself and trains directly on the preference pairs with a single classification-style loss — no explicit reward model and no PPO sampling loop.

Buy (b) preference comparisons. Reasoning tied to each stage's job:

• SFT can only imitate answers we wrote; the model is already fluent, so more demonstrations mostly repeat what it can do.
• The failure — choosing the worse of two plausible answers — is a ranking problem over the model's own outputs, exactly what preference data + the reward model fix. Comparisons give feedback on outputs we never wrote, covering the space the model actually explores.
• So more comparisons sharpen the reward model and, through PPO, steer the policy toward the better answer among plausible ones.

L2.1 loss $\approx 0.2014$ ::: L2.2 loss $\approx 1.7014$ ::: L2.3 penalty $\approx 0.2773$ L4.1 PPO term $=3.60$ ::: L4.2 PPO term $=-3.90$ ::: L4.3 adjusted reward $\approx 0.6227$