Noesis
4.4.2Alignment, Prompting & RAG

Reward modeling

1,909 words9 min readdifficulty · medium2 backlinks
🇮🇳 Yeh note Hinglish mein padho →

WHY does reward modeling exist?

WHAT is a reward model?

HOW do we turn "A beats B" into a loss? (Derivation from scratch)

We need a probabilistic model of preferences. Start from first principles.

Step 1 — Assume each response has a hidden "quality" score. Let the reward model assign rw=rθ(x,yw)r_w = r_\theta(x,y_w) and rl=rθ(x,yl)r_l = r_\theta(x,y_l). Why? We must convert two scalars into a probability that ywy_w is chosen.

Step 2 — The Bradley–Terry model. Assume the probability a human prefers ywy_w depends only on the score difference, monotonically. The standard choice: probability grows with rw−rlr_w-r_l through a logistic (sigmoid).

P(yw≻yl∣x)=erwerw+erl=σ(rw−rl)P(y_w \succ y_l \mid x) = \frac{e^{r_w}}{e^{r_w}+e^{r_l}} = \sigma(r_w - r_l)

Why the sigmoid? ere^{r} is a positive "strength". The share of strength going to the winner is erwerw+erl\tfrac{e^{r_w}}{e^{r_w}+e^{r_l}}. Dividing top and bottom by erwe^{r_w} gives 11+e−(rw−rl)=σ(rw−rl)\tfrac{1}{1+e^{-(r_w-r_l)}}=\sigma(r_w-r_l). So only the difference matters, not the absolute scale.

Step 3 — Maximize likelihood = minimize negative log-likelihood. For one example the likelihood of the observed human choice is σ(rw−rl)\sigma(r_w-r_l). Take −log⁡-\log:

Step 4 — Check the gradient makes sense. Let Δ=rw−rl\Delta = r_w-r_l. Using ddΔlog⁡σ(Δ)=1−σ(Δ)=σ(−Δ)\frac{d}{d\Delta}\log\sigma(\Delta)=1-\sigma(\Delta)=\sigma(-\Delta): ∂L∂rw=−σ(−Δ),∂L∂rl=+σ(−Δ).\frac{\partial \mathcal L}{\partial r_w} = -\sigma(-\Delta),\qquad \frac{\partial \mathcal L}{\partial r_l} = +\sigma(-\Delta). Why sensible? If the model is already confident (Δ\Delta large positive), σ(−Δ)→0\sigma(-\Delta)\to0 — tiny gradient, "nothing to fix". If it's wrong (Δ<0\Delta<0), σ(−Δ)\sigma(-\Delta) is large — big correction. Self-correcting.

Figure — Reward modeling

Worked examples

After training: how the reward model is used

Common mistakes (Steel-manned)

Flashcards

What is a reward model in RLHF?
A learned scalar function rθ(x,y)r_\theta(x,y) trained on human preferences to score how good response yy is for prompt xx; usually the base LM with a scalar output head.
Why can't we just hand-write the reward for "be helpful"?
The property is easy to recognize but impossible to specify formulaically; humans can compare outputs though, so we learn the reward from comparisons.
State the Bradley–Terry preference probability.
P(yw≻yl)=σ(rθ(x,yw)−rθ(x,yl))P(y_w\succ y_l)=\sigma(r_\theta(x,y_w)-r_\theta(x,y_l)).
Write the pairwise reward-model loss.
L=−E[log⁡σ(rθ(x,yw)−rθ(x,yl))]\mathcal L=-\mathbb E[\log\sigma(r_\theta(x,y_w)-r_\theta(x,y_l))].
Why does only the reward difference matter, not the absolute value?
The sigmoid depends only on rw−rlr_w-r_l; adding a constant to all rewards leaves the loss unchanged (scale/shift invariant).
What is the gradient w.r.t. rwr_w and why is it self-correcting?
∂L/∂rw=−σ(−Δ)\partial\mathcal L/\partial r_w=-\sigma(-\Delta); large when the model is wrong (Δ<0\Delta<0), near zero when confidently right.
How is a ranking of K responses turned into training data?
Form all (K2)\binom{K}{2} pairs and average their pairwise losses.
What is reward hacking / over-optimization?
The policy exploits regions where the RM is confidently wrong, raising measured reward while true preference drops (Goodhart).
Why include a KL penalty when optimizing against the RM?
To keep the policy near the reference distribution so the RM stays in-distribution and can't be exploited.
Why prefer pairwise comparison loss over MSE regression on ratings?
Comparisons are more reliable than absolute human scores; the logistic loss is scale-free.
Recall Feynman: explain to a 12-year-old

Imagine you're teaching a robot to write nice birthday cards, but "nice" is hard to describe. So instead you show it two cards and just point to the one you like more, again and again. From all your pointing, the robot builds a little "niceness meter" that gives every card a score. It doesn't matter if the numbers are 5 and 3 or 105 and 103 — what matters is that your favourite gets the higher one. Later the robot writes new cards trying to make the meter go up. But careful: if it only chases the meter, it might find a silly trick the meter loves but you actually hate — so we keep it on a short leash to stay sensible.

Connections

  • RLHF — reward modeling is its middle stage (SFT → RM → PPO).
  • Bradley-Terry model — statistical foundation of the pairwise loss.
  • Logistic regression — same sigmoid-NLL structure on the score difference.
  • PPO — consumes rθr_\theta as the reward signal.
  • KL divergence — the leash preventing over-optimization.
  • Goodhart's law — why a proxy reward degrades under pressure.
  • DPO Direct Preference Optimization — skips the explicit RM using the same BT likelihood.
  • Prompting and RAG — alternative ways to steer behavior without training a reward.

Concept Map

motivates

trains

cheap to collect

form

gives

scores yw and yl

negative log-likelihood

only difference matters

scalar score used in

gradient sigmoid of neg delta

Human preferences no formula

Reward model r-theta

Preference dataset x yw yl

Pairwise comparisons A beats B

Bradley-Terry model

P equals sigmoid of score diff

Pairwise RM loss

Push rw above rl

RLHF policy optimization

Gradient check

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Reward modeling ka core idea simple hai: hum chahte hain ki AI humans ki pasand ke hisaab se behave kare, lekin "achha jawab" ka koi formula nahi likha ja sakta. Toh hum humans se sirf comparison lete hain — "Answer A better hai ya B?" — aur usse ek reward model rθ(x,y)r_\theta(x,y) sikhaate hain jo har answer ko ek number (score) deta hai. Ye model actually base LM hi hota hai, bas last mein ek scalar head laga dete hain.

Ab in comparisons ko loss mein convert kaise karein? Yahan Bradley–Terry model aata hai. Maan lo winner ka score rwr_w aur loser ka rlr_l hai. Human winner ko choose karega with probability σ(rw−rl)\sigma(r_w - r_l) — sirf difference matter karta hai, absolute value nahi. Iska −log⁡-\log le lo toh loss ban jaata hai: −log⁡σ(rw−rl)-\log\sigma(r_w-r_l). Iska matlab: model ko winner ko loser se comfortably upar rakhna hai. Beauty ye hai ki agar model already sahi hai toh gradient chhota, aur agar galat hai toh gradient bada — yaani ye khud ko correct karta hai.

Ye kyun important hai? Kyunki ye RLHF ka bilkul beech ka step hai: pehle SFT, phir ye reward model, phir PPO se policy is reward ko maximize karti hai. Par ek warning: reward model perfect nahi hota. Agar policy sirf iss score ko andha-dhundh chase kare, toh wo aise tricks dhoondh legi jinko meter pasand karta hai par insaan nahi — isko reward hacking / Goodhart's law kehte hain. Isliye ek KL penalty lagate hain jo policy ko reference model ke paas rakhta hai, taaki reward model apne "comfort zone" mein rahe. Yaad rakho: reward sirf relative hai, per-prompt — absolute number ka koi matlab nahi.

Go deeper — visual, from zero

D1FoundationsEvery symbol from zero · 2,460 wordsD2Visual walkthroughThe derivation in pictures · 2,019 wordsD3Worked examplesEvery scenario covered · 2,847 wordsD4ExercisesGraded, full solutions · 2,724 wordsD5Question bankConcept traps, quick reveals · 1,658 words

Test yourself — Alignment, Prompting & RAG

L1 RecognitionL2 RecallL3 ProductionL4 ApplicationL5 Mastery

Connections

Logistic regression — sigmoid, cross-entropy lossCoding · 5.6.2Entropy and KL divergenceAI-ML · 1.3.18Direct Preference Optimization (DPO)AI-ML · 4.4.4Direct Preference Optimization (DPO)AI-ML · 4.4.4Constitutional AI overviewAI-ML · 4.4.5