Reinforcement Learning from Human Feedback (RLHF)

WHAT is RLHF?

The big idea: preferences are cheaper and more reliable than absolute scores. A human is bad at saying "this answer is a 7.3/10" but good at saying "answer A is better than answer B."

WHY not just do more supervised learning?

Stage 1 — Supervised Fine-Tuning (SFT)

WHAT: Fine-tune the base LM on a dataset of (prompt, ideal human answer) pairs with ordinary cross-entropy.

WHY: The base model predicts internet text, not helpful assistant replies. SFT gives a sane starting policy $\pi^{\text{SFT}}$ that already talks like an assistant, so RL has a decent place to start (RL from scratch on language would be hopeless).

Stage 2 — The Reward Model (derived from scratch)

WHAT: Given a prompt $x$ and two responses $y_w$ (winner) and $y_l$ (loser), a labeler picks the better one. We want a scalar model $r_\theta(x,y)$ such that better answers get higher reward.

HOW do we turn "A beats B" into a loss? We use the Bradley–Terry model of pairwise comparisons. Suppose each response has a latent "quality" $r$. Bradley–Terry says the probability a human prefers $y_w$ over $y_l$ is a logistic function of the reward difference:

$P(y_w \succ y_l \mid x) = \frac{e^{r_\theta(x,y_w)}}{e^{r_\theta(x,y_w)} + e^{r_\theta(x,y_l)}}$

Why this form? Divide top and bottom by $e^{r_\theta(x,y_w)}$... let's instead factor to reveal the sigmoid. Divide numerator and denominator by $e^{r_\theta(x,y_w)}$:

$P = \frac{1}{1 + e^{-(r_\theta(x,y_w) - r_\theta(x,y_l))}} = \sigma\big(r_\theta(x,y_w) - r_\theta(x,y_l)\big)$

Why this step? It shows the model only cares about the difference of rewards — exactly what a comparison gives us. The absolute scale is unidentified (adding a constant to all rewards changes nothing), which is fine.

Now maximize the likelihood of the observed human choices → minimize negative log-likelihood:

The RM is usually the SFT model with the final token-prediction head replaced by a single scalar head on the last token.

Stage 3 — RL optimization with PPO

WHAT: Treat the LM as a policy $\pi_\phi(y\mid x)$. Sample a response, score it with the RM, and update $\phi$ to increase expected reward.

The naive objective: $\max_\phi \; \mathbb{E}_{x,\;y\sim\pi_\phi}\big[\,r_\theta(x,y)\,\big]$

WHY this is dangerous: The RM is only accurate near data it saw. If we push hard on pure reward, the policy finds adversarial gibberish that fools the RM (reward hacking). So we add a leash back to the SFT policy.

WHY the KL term (derivation of intuition): We want high reward but stay close to a trusted model. Rewrite the per-token reward as $r'(x,y) = r_\theta(x,y) - \beta\log\frac{\pi_\phi}{\pi^{\text{SFT}}}$. If $\pi_\phi$ drifts far from $\pi^{\text{SFT}}$, the log-ratio blows up and eats the reward. This keeps generations fluent and prevents mode collapse.

HOW does PPO update the policy? PPO maximizes reward while limiting how far each update moves the policy, using the clipped surrogate objective. With ratio $\rho_t = \frac{\pi_\phi(a_t|s_t)}{\pi_{\phi_{\text{old}}}(a_t|s_t)}$ and advantage $A_t$:

$\mathcal{L}^{PPO} = \mathbb{E}\Big[\min\big(\rho_t A_t,\; \text{clip}(\rho_t, 1-\epsilon, 1+\epsilon)A_t\big)\Big]$

Why clip? A single big update on a noisy reward can destroy the policy. Clipping removes the incentive to move $\rho_t$ beyond $[1-\epsilon,1+\epsilon]$, giving small, safe steps.

Worked Example 1 — Reward-model loss numerically

Suppose for one comparison $r_\theta(x,y_w) = 2.0$ and $r_\theta(x,y_l) = 0.5$.

• Step: difference $\Delta = 2.0 - 0.5 = 1.5$. Why? RM only sees the gap.
• Step: $\sigma(1.5) = \frac{1}{1+e^{-1.5}} \approx 0.8176$. Why? Probability we predicted the winner correctly.
• Step: loss $= -\log(0.8176) \approx 0.201$. Why? Low loss because the model already ranked the winner higher; gradient will still gently widen the gap.

If instead the model was wrong ($r_w=0.5, r_l=2.0$): $\sigma(-1.5)\approx0.182$, loss $=-\log(0.182)\approx1.70$ — much bigger, pushing a strong correction. This is exactly what we want.

Worked Example 2 — The KL leash in action

Say for a token the SFT model gives probability $0.10$ to a word and the current policy gives $0.40$.

• Log-ratio $= \log(0.40/0.10) = \log 4 \approx 1.386$.
• With $\beta = 0.2$, penalty $= 0.2 \times 1.386 \approx 0.277$ subtracted from the reward.
• Why this step? The policy got 4× more confident than the trusted model on this token; the penalty says "only do that if the reward gain clearly beats $0.277$."

Common mistakes

Active-recall flashcards

#flashcards/ai-ml

Connections

• Supervised Fine-Tuning (SFT) — provides the starting policy and reference model.
• Proximal Policy Optimization (PPO) — the RL algorithm doing the update.
• KL Divergence — the leash keeping the policy near SFT.
• Bradley-Terry Model — statistical basis of the reward loss.
• Direct Preference Optimization (DPO) — RLHF without an explicit RM/RL loop.
• Reward Hacking and Goodhart's Law — the failure RLHF must guard against.
• Constitutional AI / RLAIF — replacing human labels with AI feedback.
• Prompting and Retrieval-Augmented Generation (RAG) — alternative/complementary alignment levers.

Concept Map

Hinglish (regional understanding)