Reinforcement Learning from Human Feedback (RLHF): Visual walkthrough

We will build, in order:

what a preference even is (a pointing finger),
what a reward number is (a height),
why we compare differences and not absolutes,
how the sigmoid turns a difference into a probability,
how maximum likelihood turns that probability into a loss,
and the degenerate cases (tie, already-correct, confidently-wrong).

Prerequisites we lean on: Bradley-Terry Model, Supervised Fine-Tuning (SFT) (the model whose head we reuse), and later KL Divergence / Proximal Policy Optimization (PPO) for stage 3.

Step 1 — The only raw material: a human points

WHAT. A labeler sees a prompt $x$ and two candidate answers. They pick one as better. That's it. No numbers, no scores — just a finger pointing.

We name the two answers by their fate:

$y_{w}$ — the winner (the one pointed at),
$y_{l}$ — the loser (the other one).

The symbol $≻$ means "is preferred to". So the whole datum is the sentence $y_{w} ≻ y_{l}$ given $x$ .

WHY start here. Humans are unreliable at "rate this 7.3/10" but reliable at "this one is better." We refuse to invent numbers we don't have. Our entire machine must be buildable from pointing alone.

PICTURE. Two answer cards; a hand points at the left one. There is no number anywhere yet — that is the whole point of Step 1.

Step 2 — Invent a height called "reward"

WHAT. We imagine every answer has a hidden number — a "quality height" — and we build a model $r_{θ} (x, y)$ that reads a prompt+answer and outputs that single number.

Reading the symbols:

$r$ — the reward (a scalar, one number),
$θ$ — the model's tunable weights (the knobs we will turn),
$(x, y)$ — the inputs it looks at: the prompt and one answer.

So $r_{θ} (x, y)$ = "how high this answer stands, according to a model with knobs $θ$ ."

WHY invent a height at all. A pointing finger compares two things but gives us no function we can optimize a language model against later. We need a machine that outputs a score for any single answer, so stage 3 can score answers it has never seen. The height is that bridge.

PICTURE. The two cards from Step 1, now placed at two heights on a vertical axis labeled $r_{θ}$ . Winner sits higher — but notice the axis has no fixed zero yet; we could slide both cards up together and nothing about "which is higher" would change. Remember that; it matters in Step 5.

Step 3 — Only the difference of heights is meaningful

WHAT. We compute one number: the gap between the winner's height and the loser's. $Δ = winner height r_{θ} (x, y_{w}) - loser height r_{θ} (x, y_{l})$

$Δ > 0$ → the model already ranks the winner higher (good).
$Δ = 0$ → the model thinks they're equal (undecided).
$Δ < 0$ → the model ranks the loser higher (wrong — must be corrected).

WHY the difference and not the two heights separately. The human never told us an absolute height — only "left is above right." So the only thing our data can pin down is the gap $Δ$ . Sliding both heights up by the same amount leaves $Δ$ untouched, and our human would report the exact same preference. Everything downstream must depend on $Δ$ alone.

PICTURE. A vertical bracket measuring the gap $Δ$ between the two card heights. A ghosted "slid up" copy of both cards shows the gap is identical — proving the absolute zero is invisible to the data.

Step 4 — Turn the gap into a probability with the sigmoid

WHAT. We need to convert the gap $Δ$ (any real number, $- \infty$ to $+ \infty$ ) into a probability (between $0$ and $1$ ) that the human prefers $y_{w}$ . The tool that does exactly this squashing is the sigmoid: $σ (Δ) = \frac{1}{1 + e ^{- Δ}}$

Reading each piece:

$Δ$ — the reward gap from Step 3, the input,
$e^{- Δ}$ — a positive number; big when $Δ$ is very negative, tiny when $Δ$ is very positive,
$\frac{1}{1 + e ^{- Δ}}$ — forced into $(0, 1)$ , so it reads as a probability.

WHY the sigmoid and not, say, clipping or a straight line? Three demands must all hold:

output must live in $(0, 1)$ (it's a probability),

bigger gap → higher probability (monotone),

depend on $Δ$ only (Step 3's rule).

The Bradley-Terry Model gives this shape for free. It posits $P (y_{w} ≻ y_{l}) = \frac{e ^{r_{w}}}{e ^{r_{w}} + e ^{r_{l}}}$ . Divide top and bottom by $e^{r_{w}}$ : $\frac{e ^{r_{w}}}{e ^{r_{w}} + e ^{r_{l}}} = \frac{1}{1 + e ^{- (r_{w} - r_{l})}} = σ (Δ) .$ The exponentials cancel into a pure function of the difference — nature handing us Step 3's requirement automatically.

PICTURE. The S-curve. Mark three points: $Δ = 0$ maps to $0.5$ (a coin flip — model is undecided), large positive $Δ$ hugs $1$ (sure the winner wins), large negative hugs $0$ . The horizontal axis is $Δ$ ; the vertical axis is $P (y_{w} ≻ y_{l})$ .

Step 5 — Maximum likelihood: reward being right, punish being wrong

WHAT. We have one knob-setting $θ$ that produces a probability $p = σ (Δ)$ of the human choice we actually observed. Good knobs make $p$ close to $1$ . We turn "make $p$ big" into "make a loss small" by taking the negative logarithm: $ℓ = - lo g p = - lo g σ (Δ)$

Reading it:

$p$ near $1$ → $lo g p$ near $0$ → loss $ℓ$ near $0$ (we predicted the human well: barely nudge),
$p$ near $0$ → $lo g p \to - \infty$ → loss $ℓ \to + \infty$ (we predicted the wrong answer: huge push).

WHY the log, and why negative. We want to reward the model for assigning high probability to the choice the human made. Probability multiplies across independent examples; $lo g$ turns products into sums (stable to optimize). The minus sign flips "maximize probability" into "minimize loss," so gradient descent can chew on it. This is maximum likelihood — nothing arbitrary, just "make the observed data most probable."

PICTURE. Plot $ℓ = - lo g σ (Δ)$ against $Δ$ . It is near-zero on the right (model already correct), rises gently through $Δ = 0$ where $ℓ = lo g 2 \approx 0.693$ , and shoots up on the left (model confidently wrong). The steepness on the left is the correction force.

Averaging this single-example loss over the whole dataset gives the parent's final result:

Step 6 — The degenerate cases (never leave the reader guessing)

WHAT. We check the three boundary situations the S-curve of Step 4 quietly covers.

Case A — a tie ( $Δ = 0$ ). The model scores both equally. Then $p = σ (0) = 0.5$ and $ℓ = - lo g 0.5 = lo g 2 \approx 0.693$ . This is the baseline loss: exactly what a coin-flip guesser pays. Any real learning drives loss below $0.693$ .

Case B — already correct ( $Δ = + 1.5$ ). $p = σ (1.5) \approx 0.8176$ , $ℓ \approx 0.201$ . Small loss, small gradient — the model is only gently pushed to widen an already-correct gap. (This is Worked Example 1 in the parent.)

Case C — confidently wrong ( $Δ = - 1.5$ ). $p = σ (- 1.5) \approx 0.182$ , $ℓ \approx 1.70$ . Loss is $8.5 \times$ larger than Case B — a strong corrective shove, exactly where we want the learning to concentrate.

WHY show all three. A reader must never hit a scenario the picture didn't cover: what if scores tie? what if the model's already right? what if it's badly wrong? Each maps to a specific height on the Step-5 loss curve, and each produces a sensible, self-correcting force.

PICTURE. The Step-5 loss curve again, now with three dots — green (Case B, right), gray (Case A, tie) and red (Case C, wrong) — each labeled with its $(Δ, ℓ)$ pair, so the whole behavior is one glance.

The one-picture summary

Recall Feynman: retell the walkthrough in plain words

All we ever get from a person is a finger pointing at the better of two answers. So we make up a "height" number for each answer using a model with knobs. We can't trust the actual height — only which is higher — so we look at the gap between them. A gap can be any size, but we want a probability that the person prefers the winner, so we bend the gap through an S-curve (the sigmoid): big positive gap → nearly certain, zero gap → coin flip, big negative gap → nearly certainly wrong. Then we say "make the answer the human actually picked as probable as possible," which — after taking the negative log so small is good and big is bad — becomes our loss. A tie costs $lo g 2$ ; being right costs almost nothing; being confidently wrong costs a lot. Turn the knobs to shrink that cost across every comparison, and the model learns to rank answers the way people do. That trained ranker is the reward model that stage 3's PPO later optimizes against — on a KL leash.

Recall Self-test

Why does the loss depend only on the reward difference? ::: Because the human only told us "winner > loser" — the absolute zero of reward is unobservable, so adding a constant to every reward leaves $Δ$ , the probability, and the loss unchanged. What is the loss when the model scores both answers equally? ::: $- lo g σ (0) = lo g 2 \approx 0.693$ , the coin-flip baseline. Which tool converts the unbounded gap $Δ$ into a $(0, 1)$ probability, and why that one? ::: The sigmoid — it is monotone, lands in $(0, 1)$ , and is exactly what Bradley–Terry produces after cancelling the exponentials, giving dependence on $Δ$ alone.