Foundations — Reinforcement Learning from Human Feedback (RLHF)

This page assumes you have seen nothing. Before RLHF makes sense, we must earn every symbol the parent note throws at you — $\pi$, $\sigma$, $r_{\theta }$, $\log$, $\mathbb{E}$, $D_{KL}$, and the strange word policy. We build them one at a time, each with a picture, in an order where every new symbol only uses ones already defined.

0. What is a "language model" really? — the symbol $\pi$

We give this whole guessing machine a name: $\pi$ (the Greek letter "pi", here it has nothing to do with $3.14$ — it is just the traditional letter for a policy, defined below).

• $\pi (y\mid x)$ reads: "the probability the model assigns to producing the full answer $y$, given the prompt $x$." The vertical bar $\mid$ means "given" / "assuming we already saw."
• $x$ = the input text (prompt). $y$ = a candidate answer (a whole sentence).

Related: this trained-to-chat model is what Supervised Fine-Tuning (SFT) produces, written ${\pi }^{\text{SFT}}$.

1. Probability, and why $0$ and $1$ are special

A probability is a number in the range $[0,1]$:

• $0$ means "never,"
• $1$ means "certain,"
• $0.5$ means "coin-flip."

The model spreads a total of $1.0$ across all possible next words — it is dividing up a full pie. If it hands "the" a slice of $0.4$, that word gets $40\%$ of its confidence. We will constantly compare two models' slices for the same word, so hold that picture: two pies, same slice, different sizes.

2. The logarithm $\log$ — turning "how many times bigger" into a plain number

The parent note writes $\log 4\approx 1.386$ and $-\log (0.8176)$. Why does a log appear at all?

Two facts we need, both visible in the figure:

1. $\log$ turns a ratio into a difference. $\log \frac{a}{b}=\log a-\log b$. This is why comparisons (which are naturally ratios/differences) become clean sums.
2. $\log$ punishes small probabilities hard. As a probability slides toward $0$, its $\log$ dives toward $-\infty$. So $-\log (p)$ is a "surprise meter": tiny probability = huge penalty.

3. The reward $r_{\theta }(x,y)$ — a judge that scores answers

Picture a judge at a talent show holding up a scorecard. But — crucial — this judge was never taught absolute scores. It was only ever shown pairs and told which one won. So only differences between its scores mean anything (see Section 5). This is the Bradley-Terry Model at work.

4. The sigmoid $\sigma$ — squashing any number into a probability

The parent note computes $\sigma (1.5)\approx 0.8176$. What is this $\sigma$ ("sigma")?

Why do we need this exact shape? Because the reward difference $\Delta =r_w-r_l$ can be any real number, but "probability the human prefers the winner" must live in $[0,1]$. The sigmoid is the S-curve that bridges those two worlds:

• $z=0\Rightarrow \sigma =0.5$ (a tie — no preference),
• big positive $z\Rightarrow \sigma \to 1$ (winner clearly preferred),
• big negative $z\Rightarrow \sigma \to 0$ (we got the ranking backwards).

5. Why only differences of reward matter

The Bradley-Terry Model says the probability a human prefers winner $y_w$ over loser $y_l$ is $\sigma (r_{\theta }(x,y_w)-r_{\theta }(x,y_l))$. Because only the difference appears, adding the same constant to every reward changes nothing: $(r_w+c)-(r_l+c)=r_w-r_l.$ So the reward's "zero point" is arbitrary. That is why "the RM outputs an absolute quality" is a listed mistake — it doesn't. It outputs a ruler with no marked origin, only spacing. This limitation is the seed of Goodhart's law.

6. Expectation $\mathbb{E}$ and KL divergence $D_{KL}$ — the leash

Expectation $\mathbb{E}[\cdot ]$ means "the average value, weighted by how likely each case is." When the parent writes ${\mathbb{E}}_{y\sim {\pi }_{\varphi }}[\dots ]$, the $\sim$ means "$y$ is sampled from" — i.e. "average this over answers the policy actually produces." It is just a careful word for average.

Now the leash. ${\pi }_{\varphi }$ is the policy we are training ($\varphi$ = "phi" labels its knobs, distinct from the judge's $\theta$). We want it to stay near the trusted ${\pi }^{\text{SFT}}$.

Picture two pies again (Section 1). If the new policy's slices match the trusted model's slices, KL $=0$ — no leash tension. If the new policy shouts "$0.40$!" where the trusted model whispered "$0.10$," the log-ratio $\log 4=1.386$ shows up as tension pulling it back. Deep dive: KL Divergence.

7. Putting the symbols in order

Each box uses only boxes that feed into it — that is the whole promise of "every symbol earned before use."

8. One numerical walkthrough, symbol by symbol

Reusing the parent's Worked Example 1 with our fresh definitions, so you feel each symbol fire.

• $r_{\theta }(x,y_w)=2.0$, $r_{\theta }(x,y_l)=0.5$ — the judge's two scores (Section 3).
• $\Delta =r_w-r_l=1.5$ — only the difference counts (Section 5).
• $\sigma (1.5)=\frac{1}{1+e^{-1.5}}\approx 0.8176$ — squash difference into "probability we ranked correctly" (Section 4).
• loss $=-\log (0.8176)\approx 0.2015$ — the surprise meter; low because we were right (Section 2).

If the judge had it backwards ($r_w=0.5,r_l=2.0$): $\Delta =-1.5$, $\sigma (-1.5)\approx 0.1824$, loss $=-\log (0.1824)\approx 1.702$ — a big penalty demanding a big correction. Every symbol on this page just did a job.

Equipment checklist

Cover the right side and answer out loud before revealing.