Worked examples — Reinforcement Learning from Human Feedback (RLHF)

Parent: Reinforcement Learning from Human Feedback (RLHF). This deep dive drills the numbers behind RLHF until no scenario can surprise you. We reuse only three tools you already met in the parent: the sigmoid $\sigma$, the Bradley–Terry reward-difference likelihood, and the KL leash. Every symbol below is re-earned in plain words before use.

Before we start, one shared reminder of the two atoms we lean on everywhere:

Read the three landmark points off that curve now, because every example uses them:

• $\sigma (0)=0.5$ — zero gap means a coin flip.
• $z\to +\infty \Rightarrow \sigma \to 1$ — a huge positive gap means "certainly the winner."
• $z\to -\infty \Rightarrow \sigma \to 0$ — a huge negative gap means "certainly wrong."

And the KL leash term, restated once so the word problems can cite it:

---

The scenario matrix

Every RLHF numerical question falls into one of these cells. The examples that follow each carry a tag like (Cell A) so you can see the whole space is covered.

Cell	Case class	What's special about it
A	RM loss, model already correct ($r_w>r_l$)	small positive loss
B	RM loss, model exactly tied ($r_w=r_l$, gap $=0$)	degenerate midpoint, loss $=\log 2$
C	RM loss, model wrong ($r_w<r_l$)	large loss, strong correction
D	RM limiting behaviour (gap $\to \pm \infty$)	loss $\to 0$ or $\to \infty$
E	RM scale invariance (add constant to both rewards)	loss unchanged
F	KL leash, policy more confident than SFT	positive penalty (log-ratio $>0$)
G	KL leash, policy less confident than SFT	negative penalty — a bonus (log-ratio $<0$)
H	KL leash, policy equals SFT	zero penalty
I	Combined reward-minus-penalty decision (word problem)	does the reward gain beat the leash?
J	Exam twist — accuracy vs. margin, or full sequence KL	subtle "feels right but wrong"

---

Example 1 — Model already ranked the winner higher (Cell A)

Forecast: Guess first — will the loss be big or small? The model already prefers the true winner, so we expect a small loss.

1. Gap $\Delta =r_w-r_l=2.0-0.5=1.5$. Why this step? Bradley–Terry says only the difference of rewards matters, never the absolute values. The comparison is our only data.
2. Probability of correct ranking $=\sigma (1.5)=\frac{1}{1+e^{-1.5}}\approx 0.8176$. Why this step? The sigmoid turns the reward gap into "chance we'd predict this winner." $\Delta =1.5$ sits on the rising right side of the s-curve — clearly above $0.5$.
3. Loss $=-\log \sigma (1.5)=-\log (0.8176)\approx 0.2014$. Why this step? Loss is negative log-likelihood: correct-and-confident $\Rightarrow$ small loss.

Verify: $\sigma (1.5)>0.5$ confirms the model is already right, and $0.20$ is well below the tied-case value $\log 2\approx 0.693$ — consistent with "model was correct." ✓

---

Example 2 — Exact tie: the degenerate midpoint (Cell B)

Forecast: The gap is zero — the model has no opinion. Guess: it should be penalized for not having learned the preference yet. How much?

1. Gap $\Delta =1.3-1.3=0$. Why this step? A tie is the boundary between "right" and "wrong" — worth pinning down exactly.
2. Probability $=\sigma (0)=\frac{1}{1+e^0}=\frac{1}{2}$. Why this step? Landmark point from the figure: zero gap is a coin flip.
3. Loss $=-\log (0.5)=\log 2\approx 0.6931$. Why this step? This is the reference loss — every comparison starts here before the RM learns anything.

Verify: Note $r_w=r_l=1.3$ was arbitrary; any equal pair gives the same $\log 2$. The value sits between the correct case (Ex. 1, $0.20$) and the wrong case (Ex. 3, $1.70$), exactly where a "no-opinion" answer belongs. ✓

---

Example 3 — Model got it backwards (Cell C)

Forecast: The model is confidently wrong. Guess: loss much larger than the tied $0.693$.

1. Gap $\Delta =0.5-2.0=-1.5$. Why this step? A negative gap means the model prefers the loser — the sign carries the mistake.
2. Probability of correct ranking $=\sigma (-1.5)=\frac{1}{1+e^{1.5}}\approx 0.1824$. Why this step? Negative $z$ lands on the left side of the s-curve, below $0.5$: the model thinks the true winner is unlikely.
3. Loss $=-\log (0.1824)\approx 1.7014$. Why this step? Big loss $\Rightarrow$ big gradient $\Rightarrow$ strong correction. This is the point: wrong-and-confident is punished hardest.

Verify: By the symmetry $\sigma (z)+\sigma (-z)=1$, we have $\sigma (1.5)+\sigma (-1.5)=0.8176+0.1824=1.000$. ✓ And $1.70\gg 0.20$: the wrong case is punished $\approx 8\times$ harder than the right case with the same gap size.

---

Example 4 — Limiting behaviour of the RM loss (Cell D)

Forecast: Guess the two extremes: one loss should shrink toward $0$, the other should explode.

1. Hugely right: $\sigma (10)=\frac{1}{1+e^{-10}}\approx 0.99995$, loss $=-\log (0.99995)\approx 4.54\times 10^{-5}$. Why this step? As $\Delta \to +\infty$, $e^{-\Delta }\to 0$, so $\sigma \to 1$ and $-\log 1\to 0$. The model is certain and correct — almost nothing left to learn.
2. Hugely wrong: $\sigma (-10)\approx 4.54\times 10^{-5}$, loss $=-\log (4.54\times 10^{-5})\approx 10.0$. Why this step? As $\Delta \to -\infty$, $\sigma \to 0$, and $-\log (\text{tiny})\to +\infty$. Note for large $|\Delta |$ the loss is $\approx |\Delta |$ — it grows linearly, no explosion faster than that, which is why training stays stable.

Verify: The two limits bracket the whole range: $0\le {\mathcal{L}}_{RM}<\infty$, minimum at $+\infty$ gap, unbounded above. Consistent with the s-curve saturating at both ends. ✓

---

Example 5 — Scale invariance: the zero point is arbitrary (Cell E)

Forecast: Guess before computing — should the loss move if we shift both rewards equally?

1. Shifted gap ${\Delta }^{\prime }=(2.0+100)-(0.5+100)=1.5$. Why this step? The constant cancels in the subtraction. This is the whole reason the RM output has no absolute meaning — only differences survive.
2. Loss $=-\log \sigma (1.5)\approx 0.2014$ — identical to Example 1. Why this step? Confirms the common mistake that "the RM is an absolute quality score" is false. It's a relative score.

Verify: $\sigma (102-100.5)=\sigma (1.5)=\sigma (2.0-0.5)$. Same number, same loss. ✓

---

Example 6 — KL leash: policy MORE confident than SFT (Cell F)

Forecast: The policy is more confident than the trusted model. Guess the sign of the penalty.

1. Log-ratio $\log \frac{{\pi }_{\varphi }}{{\pi }^{\text{SFT}}}=\log \frac{0.40}{0.10}=\log 4\approx 1.3863$. Why this step? Positive because the policy inflated this token's probability $4\times$ above SFT — exactly the drift the leash watches for.
2. Penalty $=\beta \times 1.3863=0.2\times 1.3863\approx 0.2773$. Why this step? This amount is subtracted from the token's reward: "only be this bold if the reward gain clearly beats $0.277$."

Verify: Sign check — policy more confident $\Rightarrow$ log-ratio $>0\Rightarrow$ positive penalty (a cost). Matches the parent note's worked example. ✓

---

Example 7 — KL leash: policy LESS confident than SFT gives a BONUS (Cell G)

Forecast: Many students assume the KL penalty is always a cost. Guess: cost or bonus here?

1. Log-ratio $\log \frac{0.10}{0.40}=\log (0.25)=-\log 4\approx -1.3863$. Why this step? Negative because the policy is less confident than SFT on this token — it pulled probability away.
2. Penalty $=0.2\times (-1.3863)\approx -0.2773$. Why this step? Subtracting a negative number adds to the reward — a small bonus for staying humble / closer-to-uniform than SFT on this token.

Verify: Exact sign-mirror of Example 6: $+0.2773$ became $-0.2773$. The per-token log-ratio can be either sign; only the expected value over the whole response, the true $D_{KL}({\pi }_{\varphi }\Vert {\pi }^{\text{SFT}})\ge 0$, is guaranteed non-negative. ✓ (See KL Divergence.)

---

Example 8 — KL leash: policy EQUALS SFT (Cell H)

Forecast: No drift. Guess the penalty.

1. Log-ratio $\log \frac{0.25}{0.25}=\log 1=0$. Why this step? Identical probabilities $\Rightarrow$ ratio $1\Rightarrow \log 1=0$.
2. Penalty $=0.2\times 0=0$. Why this step? The leash is slack when the policy agrees with the trusted model — it only bites on departures. This is the degenerate/zero case of the KL term.

Verify: $\log 1=0$ regardless of $\beta$; a fully un-drifted policy pays nothing. ✓

---

Example 9 — Combined decision: does the reward beat the leash? (Cell I — word problem)

Forecast: The bold phrasing has higher raw reward ($1.10>0.90$). Guess: does it still win after paying the KL cost?

1. Bold phrasing net value $=r-\text{penalty}=1.10-0.2773=0.8227$. Why this step? We subtract Example 6's penalty because that's the drift cost of being bold.
2. Safe phrasing net value $=0.90-0=0.90$. Why this step? Example 8's zero penalty — SFT already likes it, no leash cost.
3. Compare: $0.90>0.8227$, so the safe phrasing wins on the full objective. Why this step? The $0.20$ raw-reward lead of the bold option ($1.10-0.90$) is smaller than its $0.277$ leash cost, so the leash correctly vetoes it. This is PPO's objective in miniature.

Verify: Raw-reward gap $1.10-0.90=0.20<0.2773=$ penalty, so the bold option must lose — and $0.8227<0.90$ confirms it. The leash prevented a small reward bump from buying a big confidence drift. ✓

---

Example 10 — Exam twist: accuracy is NOT the loss (Cell J)

Forecast: Both are "correct" (positive gaps). Guess whether the mean loss is really near zero.

1. Loss 1 $=-\log \sigma (0.2)=-\log (0.5498)\approx 0.5981$. Why this step? ${\Delta }_1=0.2$ is barely positive — correct but unconfident, near the tied midpoint $\log 2\approx 0.693$. Accuracy says "right"; the loss says "barely."
2. Loss 2 $=-\log \sigma (3.0)=-\log (0.9526)\approx 0.0486$. Why this step? ${\Delta }_2=3.0$ is confidently correct — tiny loss.
3. Mean loss $=\frac{0.5981+0.0486}{2}\approx 0.3234$. Why this step? Far from zero, even at 100% accuracy — because loss rewards margin, not just the sign of the gap. This is why RM training keeps improving after accuracy plateaus.

Verify: $100\%$ accuracy but mean loss $0.32\ne 0$; and comparison 1's $0.60$ sits just under the tied bound $0.693$, exactly as "barely correct" should. The claim is false. ✓

---

Recall Self-check clozes

The RM loss depends only on the reward difference $\Delta =r_w-r_l$, never absolute values.

A tied comparison ($\Delta =0$) gives loss ::: $\log 2\approx 0.693$

Adding the same constant to both rewards changes the loss by ::: nothing (scale invariance)

When the policy is less confident than SFT on a token, the KL term acts as a ::: bonus (negative penalty)

A batch at 100% ranking accuracy can still have loss well above zero because loss rewards ::: margin, not just the sign of the gap