Reward modeling: Visual walkthrough — AI-ML deep dive

Step 1 — Two answers, two numbers

WHAT. For one prompt $x$ (the question a human asked) we have two answers. A human already told us which they liked: the winner $y_{w}$ and the loser $y_{l}$ . Our machine — the reward model $r_{θ}$ — reads each answer and spits out a single number: a score.

$r_{w} = r_{θ} (x, y_{w}), r_{l} = r_{θ} (x, y_{l})$

Reading it out loud, symbol by symbol:

$r_{θ}$ :: the "niceness meter". The little subscript $θ$ (theta) just means "its dials/settings are called $θ$ " — the numbers we will later tune.
$x$ :: the prompt (same for both answers).
$y_{w}$ , $y_{l}$ :: the winning and losing answer.
$r_{w}$ , $r_{l}$ :: two plain real numbers on a number line — that is all a "score" is.

WHY. We cannot argue about "which answer is better" mathematically until better is a number. Step 1 turns a fuzzy comparison into two dots on a line.

PICTURE. Two dots on a number line — the winner's dot sits to the right (higher) when the model is doing its job.

Step 2 — Only the gap between the dots matters

WHAT. Define the difference $Δ = r_{w} - r_{l} .$

$Δ$ (Greek "Delta") :: literally "the gap". It is the winner's score minus the loser's score.
$Δ > 0$ :: winner is ahead (good).
$Δ < 0$ :: model got it backwards (bad).
$Δ = 0$ :: the model can't tell them apart (a tie — a degenerate case we return to in Step 6).

WHY use the difference and not the two scores separately? Because a human's choice has no absolute zero. If I like card A more than card B, that judgement is unchanged whether the meter reads $(5, 3)$ or $(105, 103)$ — both have $Δ = 2$ . So the whole story must depend on $Δ$ only. Sliding both dots left or right by the same amount must change nothing. Watch the two rows in the figure: different absolute positions, same gap, and (spoiler) the same loss.

PICTURE. Two number lines stacked. Top: dots at $2$ and $0.5$ . Bottom: dots at $105$ and $103.5$ . The green "gap" bracket is identical length in both.

Step 3 — Turn the gap into a probability (why we need the S-curve)

WHAT. We want to predict the chance a human prefers the winner. A chance must live between $0$ and $1$ . But $Δ$ roams over the whole number line, $(- \infty, + \infty)$ . We need a squashing function that:

sends a huge gap to almost $1$ ("surely the winner"),
sends a huge negative gap to almost $0$ ("surely not"),
sends $Δ = 0$ to exactly $0.5$ (a coin flip),
rises smoothly and never repeats.

The function that does exactly this is the logistic sigmoid $σ$ : $P (y_{w} ≻ y_{l} ∣ x) = σ (Δ) = \frac{1}{1 + e ^{- Δ}} .$

Term by term:

$P (y_{w} ≻ y_{l} ∣ x)$ :: read " $≻$ " as is preferred to; the whole thing is "probability the human picks the winner".
$σ$ (sigma) :: our squasher.
$e^{- Δ}$ :: $e \approx 2.718$ raised to the power $- Δ$ . It is huge when $Δ$ is very negative (so the fraction $\to 0$ ) and tiny when $Δ$ is very positive (so the fraction $\to 1$ ). This single term is what bends the S.

WHY the sigmoid and not a straight line? A straight line would eventually give probabilities above $1$ or below $0$ — nonsense. The sigmoid is the smallest, smoothest tool that maps the whole line into $(0, 1)$ with a coin-flip at the middle.

PICTURE. The S-curve. Mark the three anchor points: $Δ \to - \infty$ hugs $0$ , $Δ = 0$ hits exactly $0.5$ , $Δ \to + \infty$ hugs $1$ .

Step 4 — From "probability" to "loss" (why the $- lo g$ )

WHAT. Training means: make the human's actual choice as likely as possible under our model. The likelihood of the observed choice is exactly $σ (Δ)$ . Maximizing a number is the same as minimizing its negative logarithm, so we define the loss of one example: $ℓ = - lo g σ (Δ) .$

$lo g$ :: the natural logarithm. It turns products into sums (handy when we multiply many examples' probabilities) and it explodes as its input approaches $0$ .
the minus sign :: flips "big probability = good" into "small loss = good", so we can minimize.

Behaviour of $ℓ = - lo g σ (Δ)$ :

confident & correct ( $Δ$ large $+$ ): $σ \to 1$ , $lo g 1 = 0$ , so $ℓ \to 0$ — almost no penalty.
coin flip ( $Δ = 0$ ): $σ = 0.5$ , $ℓ = - lo g 0.5 = 0.693$ — a fixed middling penalty.
confidently wrong ( $Δ$ large $-$ ): $σ \to 0$ , $- lo g (tiny) \to + \infty$ — huge penalty.

Averaged over the dataset this is the parent's headline formula: $L (θ) = - E_{(x, y_{w}, y_{l})} [lo g σ (r_{θ} (x, y_{w}) - r_{θ} (x, y_{l}))] .$

$E_{(x, y_{w}, y_{l})}$ :: the average over all preference triples in the data.

WHY $- lo g$ and not, say, $1 - σ$ ? Because a wrong-and-confident answer should hurt without limit, dragging the model back hard. Only $- lo g$ gives that unbounded penalty near probability $0$ ; a plain $1 - σ$ tops out at $1$ and lets confident mistakes off cheap.

PICTURE. Plot $ℓ$ against $Δ$ : near-flat at $0$ on the right, passing through $0.693$ at $Δ = 0$ , curving up steeply toward $+ \infty$ on the left.

Step 5 — The gradient: why the loss fixes itself

WHAT. To learn, we nudge the scores in the direction that lowers $ℓ$ . That direction is the gradient — the slope of the loss with respect to each score. A famous identity ( $\frac{d}{d Δ} lo g σ (Δ) = 1 - σ (Δ) = σ (- Δ)$ ) gives: $\frac{\partial ℓ}{\partial r _{w}} = - σ (- Δ), \frac{\partial ℓ}{\partial r _{l}} = + σ (- Δ) .$

$\partial ℓ / \partial r_{w}$ :: "if I raise the winner's score a little, how does the loss change?" It is negative, meaning raising $r_{w}$ lowers loss — so training pushes the winner up.
$\partial ℓ / \partial r_{l}$ :: positive, so training pushes the loser down.
$σ (- Δ)$ :: the shared size of the nudge — call it the error signal.

The magic is in $σ (- Δ)$ :

already right ( $Δ$ big $+$ ): $σ (- Δ) \to 0$ — tiny nudge, nothing to fix.
wrong ( $Δ < 0$ ): $σ (- Δ)$ is large — big nudge, swap the order.

So the update is self-correcting: the size of the fix equals the size of the mistake.

WHY does this shape appear? Because the loss is flat where the model is confidently right (Step 4's right tail) and steep where it is confidently wrong (the left tail). Gradient = slope, so it is automatically small where things are fine and large where they are broken.

PICTURE. The loss curve from Step 4 with tangent-line arrows: a nearly flat tangent on the confident-right side (small push), a steep tangent on the confident-wrong side (big push).

Step 6 — Edge and degenerate cases (nothing left unshown)

WHAT & WHY. A derivation is only trustworthy if the boundaries behave.

Case	$Δ$	$σ (Δ)$	Loss $ℓ$	Gradient size $σ (- Δ)$	Meaning
Confident correct	$+ \infty$	$\to 1$	$\to 0$	$\to 0$	Leave it alone
Exact tie	$0$	$0.5$	$0.693$	$0.5$	Max uncertainty, medium push
Confident wrong	$- \infty$	$\to 0$	$\to + \infty$	$\to 1$	Hardest correction
Shift both scores by $c$	unchanged	unchanged	unchanged	unchanged	Scale/shift invariance

The tie ( $Δ = 0$ ) is the only place with no "right answer"; the loss sensibly sits at the coin-flip value $- lo g 0.5 = 0.693$ and pushes with a middling $0.5$ .
The shift-invariance row is the parent's warning made visible: only differences mean anything, so the raw reward number carries no global meaning (see the [!mistake] about absolute values in the parent). This is why in RLHF we mean-center rewards before feeding PPO.

PICTURE. Three mini number-lines side by side (correct / tie / wrong) with the resulting $ℓ$ printed under each — the whole behavioural range at a glance.

Step 7 — Beyond two answers, and where this loss lives

WHAT. Given a human ranking of $K$ answers, we don't invent new math — we reuse Steps 1–5 on every ordered pair. With $K$ answers there are $(2 K)$ pairs; average their losses.

$L_{rank} = \frac{1}{( 2 K )} \sum_{pairs (w, l)} - lo g σ (r_{w} - r_{l}) .$ For $K = 4$ : $(2 4) = 6$ comparisons from one labelling effort.

WHY. One human ranking yields many training signals cheaply, and averaging tames noise in any single pair.

Downstream. The trained $r_{θ}$ then becomes the objective the policy climbs, leashed by a KL divergence penalty so it can't wander to where $r_{θ}$ is confidently wrong — that failure is Goodhart's law in action. A newer route, DPO Direct Preference Optimization, folds this very same $- lo g σ (Δ)$ loss directly into policy training, skipping the separate model. Either way, the S-curve you drew in Step 3 is the beating heart.

The one-picture summary

Everything above, as one flow: two answers → two scores → a gap $Δ$ → the sigmoid → the $- lo g$ loss → a self-correcting gradient.

Recall Feynman retelling — the whole walkthrough in plain words

A human points at the answer they like more. We give both answers a score (Step 1). We only ever care about the gap between the two scores, never the raw numbers — the same gap means the same thing whether the scores are $(5, 3)$ or $(105, 103)$ (Step 2). To turn that gap into a chance of preferring the winner, we push it through an S-shaped squasher that lives strictly between 0 and 1, is a coin-flip in the middle, and never wraps around (Step 3). We then say "make the human's real choice as likely as possible", which — after taking a negative logarithm so we can minimize — becomes our loss: tiny when we're confidently right, huge when we're confidently wrong (Step 4). The slope of that loss (the gradient) is automatically small where we're already right and big where we're wrong, so the meter fixes its own mistakes in proportion to how badly it messed up (Step 5). Ties and score-shifts behave exactly as they should (Step 6), and a ranking of many answers is just lots of these pair comparisons averaged together — later fed to a policy that climbs the meter on a short leash (Step 7).

Recall Quick self-check

The loss when $Δ = 0$ ::: $- lo g 0.5 = 0.693$ Why does raising $r_{w}$ lower the loss ::: $\partial ℓ / \partial r_{w} = - σ (- Δ)$ is negative, so increasing the winner's score decreases loss What cancels when you divide Bradley–Terry top and bottom by $e^{r_{w}}$ ::: the absolute scale $e^{r_{w}}$ , leaving only $Δ = r_{w} - r_{l}$ Number of pairs from a ranking of $K = 4$ ::: $(2 4) = 6$