Reward modeling — Hinglish notes

4.4.2 · AI-ML › Alignment, Prompting & RAG

WHY reward modeling exist karta hai?

WHAT hota hai ek reward model?

HOW karte hain hum "A beats B" ko ek loss mein convert? (Scratch se derivation)

Humein preferences ka ek probabilistic model chahiye. First principles se shuru karte hain.

Step 1 — Maano har response ka ek hidden "quality" score hota hai. Reward model assign kare $r_{w} = r_{θ} (x, y_{w})$ aur $r_{l} = r_{θ} (x, y_{l})$ . Kyun? Humein do scalars ko ek probability mein convert karna hai ki $y_{w}$ choose hoga.

Step 2 — Bradley–Terry model. Maano ki ek human ke $y_{w}$ prefer karne ki probability sirf score difference par depend karti hai, monotonically. Standard choice: probability $r_{w} - r_{l}$ ke saath ek logistic (sigmoid) ke through badhti hai.

$P (y_{w} ≻ y_{l} ∣ x) = \frac{e ^{r_{w}}}{e ^{r_{w}} + e ^{r_{l}}} = σ (r_{w} - r_{l})$

Sigmoid kyun? $e^{r}$ ek positive "strength" hai. Winner ki strength ka share $\frac{e ^{r_{w}}}{e ^{r_{w}} + e ^{r_{l}}}$ hai. Upar aur neeche $e^{r_{w}}$ se divide karne par milta hai $\frac{1}{1 + e ^{- (r_{w} - r_{l})}} = σ (r_{w} - r_{l})$ . Toh sirf difference matter karta hai, absolute scale nahi.

Step 3 — Likelihood maximize karo = negative log-likelihood minimize karo. Ek example ke liye observed human choice ki likelihood $σ (r_{w} - r_{l})$ hai. $- lo g$ lo:

Step 4 — Check karo ki gradient sensible hai. Maano $Δ = r_{w} - r_{l}$ . $\frac{d}{d Δ} lo g σ (Δ) = 1 - σ (Δ) = σ (- Δ)$ use karke: $\frac{\partial L}{\partial r _{w}} = - σ (- Δ), \frac{\partial L}{\partial r _{l}} = + σ (- Δ) .$ Kyun sensible hai? Agar model already confident hai ( $Δ$ bada positive), $σ (- Δ) \to 0$ — tiny gradient, "kuch fix nahi karna". Agar galat hai ( $Δ < 0$ ), $σ (- Δ)$ bada hai — bada correction. Self-correcting.

Worked examples

Training ke baad: reward model use kaise hota hai

Common mistakes (Steel-manned)

Flashcards

RLHF mein reward model kya hota hai?

Ek learned scalar function

r_{θ} (x, y)

jo human preferences par train hota hai taaki score kare ki response

y

prompt

x

ke liye kitna achha hai; usually base LM with a scalar output head.

"Be helpful" ke liye reward haath se kyun nahi likh sakte?

Yeh property pahchaanno mein aasaan hai lekin formulaically specify karna impossible hai; humans outputs compare kar sakte hain, toh hum reward comparisons se seekhte hain.

Bradley–Terry preference probability batao.

P (y_{w} ≻ y_{l}) = σ (r_{θ} (x, y_{w}) - r_{θ} (x, y_{l}))

Pairwise reward-model loss likho.

L = - E [lo g σ (r_{θ} (x, y_{w}) - r_{θ} (x, y_{l}))]

Sirf reward difference kyun matter karta hai, absolute value kyun nahi?

Sigmoid sirf

r_{w} - r_{l}

par depend karta hai; saare rewards mein constant add karne se loss unchanged rehta hai (scale/shift invariant).

r_{w}

ke w.r.t. gradient kya hai aur yeh self-correcting kyun hai?

\partial L / \partial r_{w} = - σ (- Δ)

; bada jab model galat ho (

Δ < 0

), near zero jab confidently sahi ho.

K responses ki ranking training data mein kaise convert hoti hai?

Saare

(2 K)

pairs banao aur unke pairwise losses average karo.

Reward hacking / over-optimization kya hai?

Policy un regions exploit karti hai jahan RM confidently galat hai, measured reward badhta hai jabki true preference girti hai (Goodhart).

RM ke against optimize karte waqt KL penalty kyun include karte hain?

Policy ko reference distribution ke paas rakhne ke liye taaki RM in-distribution rahe aur exploit na ho sake.

Pairwise comparison loss ko ratings par MSE regression se prefer kyun karte hain?

Comparisons absolute human scores se zyaada reliable hain; logistic loss scale-free hai.

Recall Feynman: 12-saal ke bachche ko samjhao

Socho tum ek robot ko achhe birthday cards likhna sikha rahe ho, lekin "achha" define karna mushkil hai. Toh uski jagah tum usse do cards dikhate ho aur bas woh wala point karte ho jo tumhe zyaada pasand hai, baar baar. Tumhari saari pointing se robot ek chhota sa "achhaai meter" banata hai jo har card ko score deta hai. Koi farak nahi padhta ki numbers 5 aur 3 hain ya 105 aur 103 — jo matter karta hai woh yeh hai ki tumhara favourite zyaada number paye. Baad mein robot naaye cards likhta hai meter ko upar le jaane ki koshish mein. Lekin dhyan raho: agar woh sirf meter ke peeche bhaage, toh shayad ek silly trick dhundh le jo meter ko bahut pasand ho lekin tum actually hate karo — isliye hum usse ek chhoti leash par rakhte hain sensible rehne ke liye.

Connections

RLHF — reward modeling iska middle stage hai (SFT → RM → PPO).
Bradley-Terry model — pairwise loss ki statistical foundation.
Logistic regression — score difference par same sigmoid-NLL structure.
PPO — $r_{θ}$ ko reward signal ke roop mein consume karta hai.
KL divergence — over-optimization rokne wali leash.
Goodhart's law — kyun proxy reward pressure mein degrade hota hai.
DPO Direct Preference Optimization — explicit RM skip karta hai same BT likelihood use karke.
Prompting aur RAG — reward train kiye bina behavior steer karne ke alternative tarike.