Question bank — Reinforcement Learning from Human Feedback (RLHF)

False. Bradley–Terry only fixes reward differences $\Delta$; adding a constant to every reward leaves the loss unchanged, so the zero point is arbitrary and cross-prompt comparisons of raw scores are meaningless.

False. The loss is $-\log \sigma (\Delta )$; since $\sigma (\Delta )<1$ for any finite gap $\Delta$, the loss is small but positive, and the gradient still gently widens the gap.

False. RLHF mostly reshapes the distribution over behaviours the base model already contained — it elicits and reprioritizes. Wrong facts baked in during pretraining largely survive.

False. The objective value rises but true quality collapses: with no leash the policy chases reward-hacking gibberish that fools the reward model and loses fluency and diversity.

Practically false. Here's the why, step by step: (1) a response is a sequence of many tokens, so the number of possible answers grows exponentially and is effectively unbounded; (2) SFT only gives a gradient signal on the exact answers we hand-wrote, leaving the vast remainder unsupervised; (3) once training starts, the model generates its own novel outputs that no author anticipated; (4) preferences let a human judge those self-generated outputs on the spot, so feedback follows the model into regions no finite demonstration set ever touched. That coverage of the model's own behaviour is what demonstrations structurally cannot provide.

False. Larger $\beta$ pulls the policy harder toward ${\pi }^{\text{SFT}}$, so it changes less and captures less reward. $\beta$ is a trade-off knob (reward vs. trustworthiness), not a quality dial.

False. Clipping removes the incentive to move the probability ratio ${\rho }_t$ outside $[1-ϵ,1+ϵ]$, producing small, safe steps so one noisy reward update can't destroy the policy.

False. The KL penalty (weight $\beta$) keeps the whole trained policy near the trusted SFT model; the PPO clip (band $\pm ϵ$) limits how far a single update step moves from the previous iterate. One is a global leash, the other a per-step speed limit.

The Bradley–Terry loss depends only on the difference $\Delta =r_{\theta }(x,y_w)-r_{\theta }(x,y_l)$. Without the loser you have no reference, the scale runs away to infinity, and you've thrown away exactly the comparison signal that made preferences reliable.

The rewards are unbounded real-valued scores, not probabilities. Only after taking the gap $\Delta =r_w-r_l$ and passing it through $\sigma (z)=1/(1+e^{-z})$ (which maps $\mathbb{R}\to (0,1)$) do we get a probability.

No — it's $\log \frac{{\pi }_{\varphi }}{{\pi }^{\text{SFT}}}$. We penalize the policy for drifting away from the trusted SFT model, so the policy sits on top; flipping it would reward drift instead.

Reward hacking is the policy exploiting blind spots in the reward model (a proxy), not labeler error. It's Goodhart's law: optimizing a proxy hard makes it stop tracking the true target.

Supervised learning has perfectly good gradients; the real reason is coverage. SFT only gets gradients on answers we hand-wrote, while RL turns "A ≻ B" judgments on the model's own samples into a gradient direction: push probability toward winners.

The RM is accurate only near data it saw. Pushing past that region finds adversarial inputs the RM mis-scores — precisely why the KL leash exists. Max reward $\ne$ max quality off-distribution.

Dividing the Bradley–Terry fraction top-and-bottom by $e^{r_w}$ gives $\sigma (\Delta )$ with $\Delta =r_w-r_l$, revealing that only the gap is identifiable — which matches exactly what a pairwise comparison gives us (nothing about absolute scale).

The reward is a judgment about the whole completed response; reading it off the final position lets the model summarize the entire answer into one score, mirroring how a human evaluates the finished reply.

The base model predicts internet text, so from-scratch RL over language would almost never stumble onto assistant-like replies to reward. SFT gives a sane starting policy ${\pi }^{\text{SFT}}$ that already talks like an assistant, so RL just refines it.

Humans are unreliable at absolute scores but consistent at comparisons. Preferences are cheaper, lower-variance, and — via Bradley–Terry — convert cleanly into a differentiable ranking loss.

A wrong ranking makes $\Delta <0$, so $\sigma (\Delta )$ is small and $-\log \sigma (\Delta )$ is large — the log blows up as the predicted probability of the correct choice approaches zero, forcing a strong corrective gradient.

Because the policy objective and the training loss both depend only on reward differences $\Delta$ along the response; a global constant cancels everywhere, so the unfixed offset never affects any decision.

$\Delta =0$, so $\sigma (0)=0.5$ and loss $=-\log 0.5\approx 0.693$ — the model is maximally unsure, and the gradient pushes to separate the two.

$\sigma (\Delta )\to 1$, so the loss $-\log \sigma (\Delta )\to 0$ and the gradient vanishes — the model is already confidently correct, so training stops pushing that pair. This is why "already correct" pairs contribute almost nothing.

$\sigma (\Delta )\to 0$, so the loss $-\log \sigma (\Delta )\to +\infty$ and its gradient grows without bound — a confidently wrong ranking produces a huge corrective push, exactly the self-correcting behaviour we want from the log.

The log-ratio $\log \frac{{\pi }_{\varphi }}{{\pi }^{\text{SFT}}}=\log 1=0$, so that token contributes no penalty — the leash only pulls when the policy has actually drifted.

Any drift is punished infinitely, so the optimum is ${\pi }_{\varphi }={\pi }^{\text{SFT}}$: RLHF becomes a no-op and the model is just the SFT model, ignoring the reward entirely.

The policy is free to chase pure reward, so it drifts toward degenerate reward-hacking outputs that maximize the proxy while losing fluency — the classic collapse the KL term prevents.

The RM will settle near $\Delta \approx 0$ for such pairs, giving $\sigma \approx 0.5$; averaged over many noisy ties this correctly tells the model these responses are of comparable quality rather than forcing a spurious ranking.

Yes — that's the Goodhart edge case: off-distribution outputs can fool the proxy reward model into high scores while real quality drops, which is exactly why we monitor the KL to ${\pi }^{\text{SFT}}$.