Intuition The one core ideaWe cannot write down a formula for "a good answer", but a human can point to the better of two answers. Reward modeling turns that pointing into a learned scoring function — a number the machine attaches to any answer — so that the answer humans prefer always gets the higher number.
This page assumes nothing . Every symbol in the parent note — r θ σ Δ E log KL ≻
Reward modeling has exactly four "nouns". Meet them first as pictures, then we'll name them.
Definition The four nouns
Prompt x Response y A pair of responses y w y l same prompt, where a human decided one is better.A score — a single number saying "how good is this answer".
Everything else on this page is just notation for these four things, or machinery for turning "human picked y w y w
Intuition What a subscriptis
A subscript is a tiny label hanging off a letter, like a name tag. y w y w y w inner". The little w
Symbol
Read it as
Picture
x "the prompt"
the question box
y "a response"
one answer bubble
y w "the winner response"
the answer the human chose
y l "the loser response"
the answer the human rejected
Why the topic needs this: the whole dataset is triples ( x , y w , y l )
≻ y w ≻ y l "y w y l . It is a comparison arrow for quality , exactly like >
≻ > Why it feels right: it looks like a fancy greater-than.
The fix: > numbers (5 > 3 ≻ things a human judged (y w ≻ y l replace ≻ >
This is the star of the show. Let's dismantle it piece by piece.
Definition What a function isA function is a machine: put something in, get exactly one thing out. Write f ( input ) = output one number .
r θ ( x , y )
r reward (the score).( x , y ) x and a response y the subscript θ theta . It stands for all the tunable knobs (the parameters ) inside the machine. Turning the knobs changes what scores come out.
So r θ ( x , y ) θ y x
θ θ below r depends on the knobs, and we are going to learn those knobs from data." Training = searching for the θ
∈ R r θ ( x , y ) ∈ R ∈ R all ordinary numbers on the number line — positive, negative, fractions, zero, everything. So the phrase just means: the output is one plain number , and it may be negative. Nothing exotic.
Why the topic needs this: the reward model is literally this function. Learning to score = learning the knobs θ
Before the loss, we need to convert "the winner scored Δ σ
A probability is a number between 0 1 0 1 0.5 chance a human prefers the winner.
Definition The sigmoid functionσ σ ( z ) = 1 + e − z 1
σ name for this S-shaped machine.e 2.718 e − z e z What it does: takes any number z − ∞ + ∞ ( 0 , 1 )
this machine and not a straight line?
We need something that (1) always lands in [ 0 , 1 ] 2 2 z = 0 0.5 z 1 z 0 1 0 why the sigmoid, not a line.
This σ Logistic regression ; a reward model's loss is a logistic loss on the score difference.
Δ Δ = r w − r l r w = r θ ( x , y w ) r l = r θ ( x , y l )
Δ > 0 right .Δ < 0 wrong .Δ = 0 σ ( 0 ) = 0.5
Intuition Why only the difference, never the raw scores?σ ( r w − r l ) only r w − r l c both scores, the difference is unchanged, so the probability is unchanged. This is the reason the parent insists "absolute reward is meaningless — only relative reward, per prompt." The picture: slide both scores up the number line together and their gap doesn't move.
log ( p ) "what power do I raise the base to, to get p In ML the base is e
log ( 1 ) = 0 log ( small p ) → − ∞ log log
− log loss : a number that is small when the model is right and large when it's wrong . The model's own probability of the correct choice is σ ( Δ ) − log 1 0 0 why − log σ ( Δ )
E E ( x , y w , y l ) [ ⋯ ] "the average of ⋯ The subscript names what we average over. It is nothing more than "add the loss for every example and divide by how many there are."
Why the topic needs it: one example is noisy; we train on the average loss across the whole dataset so the knobs θ
These appear only in the usage stage (feeding the RM into RLHF via PPO ). Meet them lightly so the parent's final equation isn't a wall of Greek.
π π policy : the AI's rule for choosing responses. π ϕ ϕ π ref frozen reference copy we compare against.
KL ( π ϕ ∥ π ref ) how far the new policy has drifted from the reference. 0 KL divergence .
β β β β
Intuition Why any leash at all?The reward model r θ proxy , not truth. Chase it too hard and the policy finds answers the RM loves but humans hate — Goodhart's law / reward hacking. The KL leash keeps the policy in the region where the RM was trained and is trustworthy. (This exact tug-of-war is what DPO Direct Preference Optimization later folds into a single loss.)
Function r_theta scores a response
Preference pair y_w beats y_l
Difference Delta = r_w minus r_l
Sigmoid squashes Delta into a probability
Bradley-Terry preference model
Minus log makes small-when-right loss
Pairwise reward-model loss
Expectation averages over all data
Use r_theta with a KL leash in RLHF
Read it top to bottom: the four nouns (top) build the score, the score builds the difference, the difference plus sigmoid builds a probability, − log
Worked example Sigmoid of a small differenceLet Δ = 1.5 σ ( 1.5 ) = 1 + e − 1.5 1 e − 1.5 ≈ 0.2231 σ ( 1.5 ) ≈ 1.2231 1 ≈ 0.8176 Reading it: the model thinks there's an ≈ 82%
Worked example The reflection identity in numbersσ ( − 1.5 ) = 1 − σ ( 1.5 ) ≈ 1 − 0.8176 = 0.1824 1 + e 1.5 1 = 1 + 4.4817 1 ≈ 0.1824 σ ( − z ) = 1 − σ ( z )
Worked example The loss for a confident-correct pairWith Δ = 1.5 = − log σ ( 1.5 ) = − log ( 0.8176 ) ≈ 0.2014
θ ϕ Why it feels right: both are Greek subscripts for parameters.
The fix: θ reward model's knobs (learned once, then frozen ). ϕ policy's knobs (learned later against the frozen reward). Two separate machines, two separate knob sets.
e − z e − z Why it feels right: they're written side by side.
The fix: e − z e raised to the power − z z = 0 e 0 = 1 z e − z 0 why σ 1
Self-test: cover the right side and answer each. If any stumps you, reread its section above.
What does the subscript w y w It means "winner" (the preferred response); it is a label, not multiplication.
What single object does r θ ( x , y ) One real number — the score of response y x θ
What does ∈ R It is an ordinary real number and may be negative.
What is the sigmoid σ ( z ) σ ( z ) = 1/ ( 1 + e − z ) ( 0 , 1 )
Why is a sigmoid used instead of a straight line to make a probability? A line would leave [ 0 , 1 ] 0.5
What does Δ Δ = r w − r l
State the reflection identity of the sigmoid. σ ( − z ) = 1 − σ ( z )
Why take − log To build a loss that is near zero when the model is confident-correct and explodes when it is wrong.
What does E ( x , y w , y l ) [ ⋅ ] Averages the quantity over all preference examples in the dataset.
What is the difference between θ ϕ θ ϕ
In one line, what does the KL term with β It leashes the policy near a frozen reference so it can't exploit regions where the reward model is wrong.
Recall One-breath summary of the foundationsFour nouns (prompt x y y w y l r θ θ Δ = r w − r l σ − log E r θ π ϕ β
