Reward modeling: Worked examples — AI-ML deep dive

First, the symbols we reuse (all from the parent, restated so nothing is assumed):

Definition The core quantities on every line below

$x$ = the prompt (the question or instruction). $y$ = a candidate answer to that prompt.
$r_{θ} (x, y)$ = the reward model: a learned function (the letter $θ$ just names its internal weights) that reads a prompt-and-answer pair and outputs a single real number — "how good is answer $y$ for prompt $x$ ". Everywhere below we shorten $r_{θ} (x, y_{w})$ to $r_{w}$ and $r_{θ} (x, y_{l})$ to $r_{l}$ .
$r_{w}$ = reward the model gives the human's preferred answer ("winner"). A plain real number.
$r_{l}$ = reward the model gives the rejected answer ("loser").
$Δ = r_{w} - r_{l}$ = the margin. Positive means the model ranks correctly; negative means it is wrong.
$σ (z) = \frac{1}{1 + e ^{- z}}$ = the sigmoid, a curve that squashes any real number into $(0, 1)$ . It is the model's stated probability that the winner beats the loser. $σ (0) = 0.5$ (a coin flip), $σ (+ \infty) \to 1$ , $σ (- \infty) \to 0$ .
$lo g$ everywhere on this page means the natural logarithm ( $lo g_{e}$ , base $e \approx 2.718$ ). This is the standard convention in machine learning; a base- $2$ log would just rescale every loss by the constant $lo g 2 \approx 0.693$ and change nothing about which answer wins.
Loss for one pair $= - lo g σ (Δ)$ . Small = good, big = bad.

Formula The slope you will read off the curve — derived once, used ten times

We need the derivative of the loss with respect to the margin $Δ$ , because it is the gradient that moves the two scores. Write $L (Δ) = - lo g σ (Δ)$ . Step 1 — a fact about the sigmoid. $\frac{d}{d Δ} σ (Δ) = σ (Δ) (1 - σ (Δ))$ . (Differentiate $σ = (1 + e^{- Δ})^{- 1}$ by the chain rule; the $e^{- Δ}$ that appears regroups into exactly $σ (1 - σ)$ .) Step 2 — chain rule on the log. $\frac{d}{d Δ} lo g σ (Δ) = \frac{1}{σ ( Δ )} \cdot σ (Δ) (1 - σ (Δ)) = 1 - σ (Δ)$ . Step 3 — a symmetry of the sigmoid. $1 - σ (Δ) = σ (- Δ)$ (because $σ (Δ) + σ (- Δ) = 1$ ). So $\frac{\partial L}{\partial Δ} = - (1 - σ (Δ)) = - σ (- Δ) .$ Step 4 — spread it onto the two scores. Since $Δ = r_{w} - r_{l}$ , the chain rule gives $\frac{\partial Δ}{\partial r _{w}} = + 1$ and $\frac{\partial Δ}{\partial r _{l}} = - 1$ , hence $\frac{\partial L}{\partial r _{w}} = - σ (- Δ), \frac{\partial L}{\partial r _{l}} = + σ (- Δ) .$ The two gradients are equal and opposite: whatever push raises $r_{w}$ lowers $r_{l}$ by the same amount. Their shared size $σ (- Δ)$ is the number we read below.

How to read Figure s01. The horizontal axis is the margin $Δ = r_{w} - r_{l}$ ; the vertical axis is the per-pair loss $- lo g σ (Δ)$ (both labelled on the plot). The blue curve is the loss. Trace it left-to-right: it slides down as $Δ$ grows, so a bigger correct margin always costs less. Three marked dots are the cases we work below — green at $Δ = 3$ (confidently right, loss $\approx 0.05$ ), gray at $Δ = 0$ (a tie, loss $= lo g 2 \approx 0.69$ ), red at $Δ = - 2$ (wrong, loss $\approx 2.13$ ). The orange dashed line is the straight asymptote $- Δ$ that the loss hugs on the far left, showing the penalty grows without bound when the model is confidently wrong. The steepness of the blue curve at any point is exactly the gradient magnitude $σ (- Δ)$ derived above — flat on the right (nothing to fix), steep on the left (big fix).

The scenario matrix

Every reward-modeling case falls into one of these cells. The examples that follow each tag which cell(s) they hit.

Cell	Case class	What is special	Example
A	Model confidently right ( $Δ ≫ 0$ )	tiny loss, tiny gradient	Ex 1
B	Model wrong ( $Δ < 0$ )	large loss, large gradient	Ex 2
C	Tie / degenerate ( $Δ = 0$ )	loss $= lo g 2$ , exactly $0.5$ prob	Ex 3
D	Negative rewards ( $r_{w}, r_{l} < 0$ )	only the difference matters	Ex 4
E	Shift invariance (add $c$ to both)	loss unchanged — a proof-by-example	Ex 5
F	Limiting behaviour ( $Δ \to \pm \infty$ )	loss $\to 0$ or $\to \infty$	Ex 6
G	$K$ -way ranking ( $K > 2$ )	$(2 K)$ pairs averaged	Ex 7
H	Real-world word problem	pick winner from raw scores, batch loss	Ex 8
I	RL usage / KL penalty	reward vs. distance-from-reference trade-off	Ex 9
J	Exam twist: Goodhart	measured reward up, true preference down	Ex 10

The worked examples

Worked example Ex 1 — Cell A: confidently right

$r_{w} = 3.0$ , $r_{l} = 0.0$ . Find the loss and the full gradient vector $(\partial L / \partial r_{w}, \partial L / \partial r_{l})$ .

Forecast: the model is clearly right, so guess — is the loss near $0$ or near $1$ ? Is the gradient big or tiny?

Margin $Δ = r_{w} - r_{l} = 3.0 - 0.0 = 3.0$ . Why this step? The loss and gradient depend only on $Δ$ , so compute it first.
$σ (3.0) = \frac{1}{1 + e ^{- 3}} = \frac{1}{1 + 0.0498} = 0.9526$ . Why this step? This is the model's probability the winner wins; the loss is $- lo g$ (natural log) of it.
Loss $= - lo g (0.9526) = 0.0486$ . Why this step? Reading the loss curve at $Δ = 3$ — near the flat right tail.
Gradient: using the boxed result, $\partial L / \partial r_{w} = - σ (- 3) = - 0.0474$ and $\partial L / \partial r_{l} = + σ (- 3) = + 0.0474$ . Why this step? Magnitude $0.0474$ is tiny — "almost nothing to fix", the self-correcting property; the two entries are equal and opposite.

Verify: $σ (3) + σ (- 3) = 1$ ? $0.9526 + 0.0474 = 1.000$ ✓. Tiny loss + tiny opposite gradients = confidently-right cell, as forecast.

Worked example Ex 2 — Cell B: the model is wrong

$r_{w} = 0.5$ , $r_{l} = 2.5$ (it scored the rejected answer higher). Loss and full gradient?

Forecast: wrong ordering — will the loss exceed $lo g 2 \approx 0.693$ ?

$Δ = 0.5 - 2.5 = - 2.0$ . Why? Negative margin is the fingerprint of a mistake.
$σ (- 2.0) = \frac{1}{1 + e ^{2}} = \frac{1}{1 + 7.389} = 0.1192$ . Why? The model says the winner has only a $12%$ chance — badly miscalibrated against the human label.
Loss $= - lo g (0.1192) = 2.127$ . Why? Far up the steep left arm of the loss curve — much bigger than the tie value $0.693$ , so yes, forecast holds.
Gradient: $\partial L / \partial r_{w} = - σ (- Δ) = - σ (2.0) = - 0.8808$ and $\partial L / \partial r_{l} = + σ (2.0) = + 0.8808$ . Why? Big magnitude $0.88$ → a strong shove that raises $r_{w}$ and lowers $r_{l}$ by the same amount, swapping the order.

Verify: gradient magnitude $σ (2) = 0.8808$ and Ex-1's $σ (- 3) = 0.0474$ : the wrong case pushes ~18× harder. Size of the fix tracks size of the error ✓.

Worked example Ex 6 — Cell F: limiting behaviour

What happens to the loss as $Δ \to + \infty$ and as $Δ \to - \infty$ ? Check at $Δ = 8$ and $Δ = - 8$ .

Forecast: trace the loss curve's two tails from the figure.

$Δ = 8$ : $σ (8) = 0.99966$ , loss $= - lo g (0.99966) = 0.000335$ . Why? As $Δ \to + \infty$ , $σ \to 1$ and loss $\to 0$ — a flat floor. The model can never be "over-rewarded" for being right; the loss just plateaus.
$Δ = - 8$ : $σ (- 8) = 0.000335$ , loss $= - lo g (0.000335) = 8.0003$ . Why? As $Δ \to - \infty$ , loss grows roughly linearly like $- Δ$ (since $- lo g σ (Δ) \approx - Δ$ for very negative $Δ$ ). No cap on the penalty for being confidently wrong — this is the orange dashed asymptote in Figure s01.

Verify: for large negative $Δ$ , $- lo g σ (Δ) \approx - Δ$ : at $Δ = - 8$ we get $8.0003 \approx 8$ ✓. Asymmetric curve: soft floor on the right, unbounded ramp on the left.

Worked example Ex 7 — Cell G: a 4-way ranking

A human ranks four answers best→worst: $a ≻ b ≻ c ≻ d$ , with model scores $r_{a} = 2, r_{b} = 1, r_{c} = 0.5, r_{d} = - 1$ . Compute the averaged ranking loss.

Forecast: how many pairs, and will the loss be small (rankings mostly obeyed)?

Number of pairs $= (2 4) = 6$ : $(a, b), (a, c), (a, d), (b, c), (b, d), (c, d)$ . Why? Every ordered "better beats worse" pair is a training signal — reuses one label six times (InstructGPT trick).
Margins: $Δ_{ab} = 1, Δ_{a c} = 1.5, Δ_{a d} = 3, Δ_{b c} = 0.5, Δ_{b d} = 2, Δ_{c d} = 1.5$ . Why? Each margin is winner-minus-loser using the ranking's direction.
Per-pair losses $- lo g σ (Δ)$ : $0.3133, 0.2014, 0.0486, 0.4741, 0.1269, 0.2014$ . Why? Read each off the same loss curve.
Average $= \frac{0.3133 + 0.2014 + 0.0486 + 0.4741 + 0.1269 + 0.2014}{6} = \frac{1.3657}{6} = 0.2276$ . Why? Averaging (not summing) keeps the per-prompt loss scale independent of $K$ .

Verify: average of the six pair losses $= 0.2276$ ✓. The tightest pair $(b, c)$ with margin $0.5$ contributes the most ( $0.4741$ ) — the ranking is weakest there, exactly where gradient will push hardest.

Worked example Ex 8 — Cell H: real-world word problem

A support-bot RM must rank two replies to "reset my password". Reply A (steps + link) scores $r = 1.8$ ; Reply B (just "google it") scores $r = - 0.4$ . The human preferred A. (i) Does the RM agree? (ii) What loss does this pair contribute?

Forecast: the helpful reply scored higher — the RM should agree, small loss.

Winner $=$ A (human's choice), so $r_{w} = 1.8$ , $r_{l} = - 0.4$ . Why? The winner label comes from the human, not from which score is bigger; we then check whether the model matched.
RM's own ranking: $1.8 > - 0.4$ , so the RM ranks A above B — it agrees. Why? Agreement means positive margin, low loss.
$Δ = 1.8 - (- 0.4) = 2.2$ ; $σ (2.2) = 0.9002$ ; loss $= - lo g (0.9002) = 0.1051$ . Why? A modest, healthy loss — the RM is right but not maximally confident, leaving useful gradient.

Verify: $Δ = 2.2$ , loss $= 0.1051$ ✓, and the RM's argmax score matches the human's pick ✓. This is one row of a training batch for the support bot's RM (used later by PPO / RLHF).

Worked example Ex 9 — Cell I: RL objective with a KL leash

After the reward model $r_{θ} (x, y)$ (defined at the top of this page) is trained, RLHF tunes the policy to maximize $r_{θ} (x, y) - β KL$ , where $KL$ is the KL divergence measuring how far the policy has drifted from a fixed reference model, and $β$ is a knob controlling how hard we penalize that drift.

(A note on units: KL divergence here is measured in nats — the natural unit of information you get when you use the natural log $lo g_{e}$ . One nat is just "how surprised you are, measured with base- $e$ logs"; a bigger KL in nats means the two distributions disagree more. We only ever compare KL values, so the unit is a label, not a quantity to worry about.)

Suppose for a candidate answer $r_{θ} (x, y) = 3.0$ but that answer has moved the policy so far from the reference that $KL = 5.0$ nats. With penalty weight $β = 0.2$ , is this answer a net gain over a "safe" answer with $r_{θ} (x, y) = 2.0$ and $KL = 0.5$ ?

Forecast: the risky answer has higher raw reward — but the leash might cancel that.

Risky answer objective $= r_{θ} - β KL = 3.0 - 0.2 \times 5.0 = 3.0 - 1.0 = 2.0$ . Why? The KL term converts "distance from the sensible reference model" into a reward penalty, so wandering too far costs points.
Safe answer objective $= 2.0 - 0.2 \times 0.5 = 2.0 - 0.1 = 1.9$ . Why? Small KL → tiny penalty.
Compare: $2.0 > 1.9$ , so the risky answer is barely preferred here. Why? The margin is only $0.1$ — the leash has nearly erased the risky answer's $1.0$ raw-reward lead.
Now turn the knob: raise $β$ to $0.25$ . Risky objective $= 3.0 - 0.25 \times 5.0 = 3.0 - 1.25 = 1.75$ , while the safe objective $= 2.0 - 0.25 \times 0.5 = 2.0 - 0.125 = 1.875$ . Now $1.75 < 1.875$ , so the safe answer wins. Why? A larger $β$ shortens the leash: the more we distrust the reward model, the more we force the policy to stay near the reference and refuse far-flung answers.

Verify: $3.0 - 1.0 = 2.0$ and $2.0 - 0.1 = 1.9$ , difference $= 0.1$ ✓; at $β = 0.25$ , $1.75 < 1.875$ so the ordering flips ✓. The KL weight $β$ literally sets how far the policy may wander to chase reward.

Worked example Ex 10 — Cell J: the exam twist (Goodhart / over-optimization)

As training proceeds the measured RM reward on the policy climbs: $r = 1.0, 2.0, 3.0, 4.0$ at steps $1, 2, 3, 4$ . But a gold human eval gives true preference-win-rate $= 0.60, 0.72, 0.68, 0.55$ . At which step should we stop, and what is this failure called?

Forecast: measured reward only goes up — but does true quality?

Measured reward is monotonically increasing (1→4). Why? The optimizer is doing its job: it maximizes the proxy $r_{θ}$ .
True win-rate peaks at step $2$ ( $0.72$ ), then falls to $0.68$ , $0.55$ . Why? Past step 2 the policy is exploiting regions where the RM is confidently wrong — Goodhart's law: "when a measure becomes a target, it ceases to be a good measure."
Optimal stopping step $= 2$ (highest true preference). Why? Early stopping + the KL penalty (Ex 9) are the two standard defenses against this over-optimization gap.

Verify: $ar g max (0.60, 0.72, 0.68, 0.55) =$ step $2$ ✓; measured reward at that step $= 2.0$ while later reward $4.0$ is higher yet worse — the signature of reward hacking ✓.

Recall Quick self-test across the matrix

Model gives $r_{w} = 1, r_{l} = 1$ — what loss? ::: $lo g 2 \approx 0.693$ (a tie, cell C). Add $50$ to both rewards in any pair — loss changes by? ::: Zero — shift invariance (cell E). $Δ = - 8$ loss ≈ ? and why unbounded? ::: $\approx 8$ ; for large negative $Δ$ , $- lo g σ (Δ) \approx - Δ$ grows linearly (cell F). Ranking of $K = 4$ gives how many pairwise losses? ::: $(2 4) = 6$ (cell G). What is $\partial L / \partial r_{l}$ in terms of the margin? ::: $+ σ (- Δ)$ — equal and opposite to $\partial L / \partial r_{w}$ . What is a "nat" of KL divergence? ::: The unit of information from using natural (base- $e$ ) logs — a bigger KL in nats means the policy strayed further from the reference. Measured reward rising while true preference falls is called? ::: Reward hacking / over-optimization (Goodhart, cell J).

Related: Bradley-Terry model (where $σ (Δ)$ comes from), Logistic regression (same loss shape), DPO Direct Preference Optimization (folds the RM step away), and the Hinglish walkthrough 4.4.02 Reward modeling (Hinglish).