Reward modeling: Exercises — AI-ML deep dive

This page is a self-test ladder for reward modeling. Work each problem before opening its solution. The rungs climb from "do you recognize the pieces?" up to "can you build the whole thing?".

Everything you need was built in the parent note. Keep these symbols in mind:

Definition The characters in every problem

$x$ — a prompt (the question or instruction). $y$ — a candidate answer to that prompt.
$y_{w}$ — the winner: the answer a human preferred. $y_{l}$ — the loser: the answer the human rejected, for the same prompt $x$ . (Think "w = won, l = lost".)
$r_{θ} (x, y) \in R$ — the reward model: a learned number saying "how good is answer $y$ to prompt $x$ ". Bigger = the model thinks it's better. The subscript $θ$ just means "it has tunable dials (weights)".
Shorthand for the two scores we compare: $r_{w} \equiv r_{θ} (x, y_{w})$ (winner's score) and $r_{l} \equiv r_{θ} (x, y_{l})$ (loser's score).
$L$ — the loss: a single non-negative number measuring "how unhappy we are" with the model on an example. Training minimizes it. In this whole page $L = - lo g σ (r_{w} - r_{l})$ (the pairwise reward-model loss from the parent note).
$σ (z) = \frac{1}{1 + e ^{- z}}$ — the sigmoid: a machine that squashes any real number $z$ into a probability between $0$ and $1$ . We feed it the score difference $Δ = r_{w} - r_{l}$ .
$lo g$ everywhere on this page means the natural logarithm (base $e \approx 2.718$ ), written $ln$ in some books. This is the standard choice in machine learning and is what pairs cleanly with $e$ inside the sigmoid.

The picture below (Figure 1) is the only geometry you need — glance at it before starting.

Level 1 — Recognition

Recall Solution 1.1

Answer: (b). We maximize the likelihood of the human's observed choice, whose probability is $σ (r_{w} - r_{l})$ . Maximizing a probability = minimizing its negative (natural) log, giving $L = - lo g σ (r_{w} - r_{l})$ .

(a) is MSE regression — wrong tool (see the L1 mistake below).
(c) is the probability itself, not a loss (we want something to minimize, and this grows as we get better).
(d) throws away the sign of $Δ$ , so it can't tell "right order" from "wrong order".

Recall Solution 1.2

$Δ = r_{w} - r_{l} = 0$ , so $P = σ (0) = \frac{1}{1 + e ^{0}} = \frac{1}{2} = 0.5$ . Meaning: the model has no opinion — it thinks the two answers are equally good, a coin flip. This is the exact centre of the curve in Figure 1.

Level 2 — Application

Recall Solution 2.1

Step — form the difference (WHY: only $Δ$ enters the loss). $Δ = 1.2 - (- 0.3) = 1.5$ . Step — sigmoid. $σ (1.5) = \frac{1}{1 + e ^{- 1.5}} = \frac{1}{1 + 0.2231} = 0.8176$ . Step — negative log (natural). $L = - lo g (0.8176) = 0.2014$ . On Figure 1: $Δ = 1.5$ sits well to the right, $σ$ near the top, loss small — the model already ranks the winner clearly higher, so there's little to fix.

Recall Solution 2.2

$Δ = - 0.3 - 1.2 = - 1.5$ . $σ (- 1.5) = \frac{1}{1 + e ^{1.5}} = \frac{1}{1 + 4.4817} = 0.1824$ . $L = - lo g (0.1824) = 1.7014$ . Comment: same magnitude of gap but the sign flipped, and the loss jumped from $0.20$ to $1.70$ . On Figure 1, $Δ = - 1.5$ is to the left of centre where $σ$ is small — the model disagrees with the human and is punished hard.

Recall Solution 2.3

(a) Shift. New scores: $r_{w}^{'} = 101.2, r_{l}^{'} = 99.7$ . The loss only sees the difference: $Δ^{'} = r_{w}^{'} - r_{l}^{'} = 101.2 - 99.7 = 1.5 = Δ.$ So $L = - lo g σ (1.5) = 0.2014$ , identical to 2.1. The $+ 100$ cancels: the reward model is shift-invariant. (b) Scale. Doubling gives $r_{w}^{''} = 2.4, r_{l}^{''} = - 0.6$ , so $Δ^{''} = 2.4 - (- 0.6) = 3.0 \neq = 1.5$ . Then $L = - lo g σ (3.0) = - lo g (0.9526) = 0.0486$ — different from 2.1. Punchline: the loss is invariant to shifting both rewards but not to scaling them, because scaling stretches the difference $Δ$ (here $1.5 \to 3.0$ ) and $σ$ is not scale-free in its argument. So absolute values are meaningless, but the spread of rewards genuinely matters for training.

Level 3 — Analysis

Recall Solution 3.1

Derivation (WHY each step). Two facts about the sigmoid we'll use:

$\frac{d}{d z} σ (z) = σ (z) (1 - σ (z))$ — the standard sigmoid derivative.
Therefore $\frac{d}{d z} lo g σ (z) = \frac{σ ( z ) ( 1 - σ ( z ))}{σ ( z )} = 1 - σ (z) = σ (- z)$ , using the identity $1 - σ (z) = σ (- z)$ . Now apply the chain rule. Since $Δ = r_{w} - r_{l}$ , we have $\frac{\partial Δ}{\partial r _{w}} = + 1$ . Then $\frac{\partial L}{\partial r _{w}} = \frac{\partial}{\partial r _{w}} [- lo g σ (Δ)] = - (1 - σ (Δ)) \cdot \frac{\partial Δ}{\partial r _{w}} = - σ (- Δ) \cdot (1) = - σ (- Δ) .$ So the asserted formula $\frac{\partial L}{\partial r _{w}} = - σ (- Δ)$ follows directly from the sigmoid derivative + chain rule. (i) $Δ = + 3$ : $∣ grad ∣ = σ (- 3) = \frac{1}{1 + e ^{3}} = \frac{1}{1 + 20.09} = 0.0474$ . (ii) $Δ = - 3$ : $∣ grad ∣ = σ (3) = \frac{1}{1 + e ^{- 3}} = 0.9526$ . Interpretation: when the model is already confidently correct, the update is tiny ( $0.047$ ) — "nothing to fix". When it is confidently wrong, the update is near-maximal ( $0.953$ ) — a big shove to swap the order. The loss is self-correcting: effort flows to the examples that are actually wrong.

Recall Solution 3.2

Notation reminder. $(k n)$ (read " $n$ choose $k$ ") counts how many unordered ways you can pick $k$ items out of $n$ ; its formula is $(k n) = \frac{n !}{k ! ( n - k )!}$ . Here we pick $2$ responses out of $5$ to compare. $(2 5) = \frac{5 !}{2 ! 3 !} = \frac{5 \cdot 4}{2} = 10 comparisons .$ Why all pairs, not just adjacent: one expensive human ranking yields 10 training signals instead of 4, and comparing far-apart ranks (rank 1 vs rank 5) gives a large clean margin, so the RM doesn't overfit to a single noisy neighbour comparison. This is InstructGPT's trick.

Recall Solution 3.3

Average loss is $3.15/4 = 0.7875$ for both — identical by that metric. But Model A nails 3 pairs (loss $0.05$ ↔ $σ = e^{- 0.05} = 0.9512$ ) and catastrophically fails one ( $3.0$ ↔ $σ = e^{- 3} = 0.0498$ , i.e. it confidently ranks the wrong answer higher). Model B is mildly unsure everywhere ( $σ = e^{- 0.7875} = 0.4550$ , just below a coin flip). Trust Model B more for safe deployment: it never confidently inverts a preference. Model A has a "confidently wrong" hole — exactly the kind of region a policy will find and exploit during RL. Average loss hides this because it can't see the distribution of errors.

Level 4 — Synthesis

Recall Solution 4.1

Objective per answer = reward $- β \cdot KL$ . Assume $G$ is near the reference so its KL $\approx 0$ .

$G$ : $4 - β \cdot 0 = 4$ .
$H$ : $9 - β \cdot 8$ . At $β = 1$ : $H$ scores $9 - 8 = 1 < 4$ , so the objective prefers $G$ — the KL leash defeats the hack. Threshold: indifference when $9 - 8 β_{⋆} = 4 \Rightarrow 8 β_{⋆} = 5 \Rightarrow β_{⋆} = \frac{5}{8} = 0.625$ . For $β < 0.625$ the leash is too loose and the policy chases the hacked answer $H$ ; for $β > 0.625$ it stays with $G$ . This is why $β$ exists and why it must be tuned — see KL divergence and PPO.

Recall Solution 4.2

Match the two shapes. Both are $- lo g σ (score_{w} - score_{l})$ — the only thing that changes between them is what we plug in as the "score".

In the RM loss the score is the network output $r_{θ} (x, y)$ .
In DPO the score is the same Bradley–Terry "score" role, just written differently — call it $\overset{r}{^} (x, y)$ to keep the parallel explicit. It is not a new concept; it is what replaces $r_{θ}$ inside the identical loss template: $\overset{r}{^} (x, y) = β lo g \frac{π _{ϕ} ( y ∣ x )}{π _{ref} ( y ∣ x )} (how much more probable the policy makes y than the reference, scaled by β) .$ Substitute: $\overset{r}{^}_{w} - \overset{r}{^}_{l}$ is exactly the argument of $σ$ in the DPO loss above. So DPO is Bradley–Terry with the score baked into the policy's log-ratio — same statistical model, different parameterization of the score.

Level 5 — Mastery

Recall Solution 5.1

True-preference peak — set its derivative to zero: $\frac{d}{d t} (t - 0.1 t^{2}) = 1 - 0.2 t = 0 \Rightarrow t_{⋆} = 5$ . Peak true value: $R_{true} (5) = 5 - 0.1 \cdot 25 = 5 - 2.5 = 2.5$ . Measured reward at that same $t$ : $R_{meas} (5) = 0.55 \cdot 5 = 2.75$ , and it keeps climbing for $t > 5$ (e.g. $R_{meas} (8) = 4.4$ ) even while $R_{true} (8) = 8 - 6.4 = 1.6$ has already fallen below its peak. Lesson: past $t_{⋆}$ the two curves diverge — the RM's measured reward rises while true preference drops (Goodhart's law). Stop early (around $t_{⋆}$ ) and use the KL leash; "maximize the proxy forever" destroys the thing you actually cared about. See Figure 2 below.

Recall Solution 5.2

Collect preferences. For each prompt, sample $K = 4$ answers, have humans rank them → $(2 4) = 6$ pairs per prompt (efficient labeling). Guard: annotator disagreement → use comparisons, not absolute scores.
Train the reward model with the pairwise loss $L = - lo g σ (r_{w} - r_{l})$ (Bradley-Terry model, fit like Logistic regression). Guard: confidently-wrong pockets → track pairwise accuracy, not just mean loss (Ex 3.3).
Normalize rewards per prompt before RL. Concretely: mean-center (subtract the batch mean so the average reward is $0$ ) and whiten (also divide by the batch standard deviation so the spread is $1$ , i.e. $\tilde{r} = \frac{r - mean}{std}$ ). Guard: shift nonsense (Ex 2.3a) and unstable step sizes from an arbitrary reward scale (Ex 2.3b).
Optimize the policy against $r_{θ}$ with PPO + a KL penalty $β$ (Ex 4.1), or skip 2–4 and use DPO directly (Ex 4.2). Guard: reward hacking → tune $β$ , early-stop at the Goodhart peak (Ex 5.1).
Deploy with the aligned policy; the input side is shaped by Prompting and, for factual grounding, RAG. Guard: distribution shift at deploy → keep monitoring and re-collect preferences.

Recall Solution 5.3

Per-pair losses $L = - lo g σ (Δ)$ (natural log):

$Δ = 2.0$ : $σ (2) = 0.8808$ , loss $= - lo g (0.8808) = 0.1269$ .
$Δ = 0.0$ : $σ (0) = 0.5$ , loss $= - lo g (0.5) = 0.6931$ .
$Δ = - 1.0$ : $σ (- 1) = 0.2689$ , loss $= - lo g (0.2689) = 1.3133$ . Mean $= \frac{0.1269 + 0.6931 + 1.3133}{3} = \frac{2.1333}{3} = 0.7111$ . The negative- $Δ$ pair dominates — consistent with the self-correcting behaviour of Level 3.

Recall One-line summary of the whole ladder

Everything above is one formula seen from five heights: $L = - lo g σ (r_{w} - r_{l})$ — recognize it (L1), plug numbers in (L2), read its gradient and error tails (L3), connect it to PPO/DPO (L4), and respect its Goodhart ceiling (L5).