Reinforcement Learning from Human Feedback (RLHF)

4.4.1 · AI-ML › Alignment, Prompting & RAG

RLHF KYA HAI?

Badi idea: preferences absolute scores se sasti aur zyada reliable hoti hain. Ek human "yeh answer 7.3/10 hai" kehne mein bura hai lekin "answer A, answer B se behtar hai" kehne mein achha hai.

Sirf supervised learning aur kyun nahi karein?

Figure — Reinforcement Learning from Human Feedback (RLHF)

Stage 1 — Supervised Fine-Tuning (SFT)

KYA: Base LM ko (prompt, ideal human answer) pairs ke dataset par ordinary cross-entropy ke saath fine-tune karo.

KYUN: Base model internet text predict karta hai, helpful assistant replies nahi. SFT ek sane starting policy $π^{SFT}$ deta hai jo already assistant ki tarah baat karta hai, taaki RL ke paas shuru karne ki ek decent jagah ho (language par scratch se RL hopeless hota).

Stage 2 — Reward Model (scratch se derived)

KYA: Ek prompt $x$ aur do responses $y_{w}$ (winner) aur $y_{l}$ (loser) diye gaye, ek labeler behtar choose karta hai. Hum chahte hain ek scalar model $r_{θ} (x, y)$ jisme behtar answers ko zyada reward mile.

"A beats B" ko loss mein kaise badlein? Hum pairwise comparisons ka Bradley–Terry model use karte hain. Maano har response ki ek latent "quality" $r$ hai. Bradley–Terry kehta hai ki probability ki ek human $y_{w}$ ko $y_{l}$ par prefer kare wo reward difference ka ek logistic function hai:

$P (y_{w} ≻ y_{l} ∣ x) = \frac{e ^{r_{θ} (x, y_{w})}}{e ^{r_{θ} (x, y_{w})} + e ^{r_{θ} (x, y_{l})}}$

Yeh form kyun? Top aur bottom ko $e^{r_{θ} (x, y_{w})}$ se divide karo... chaliye instead sigmoid reveal karne ke liye factor karte hain. Numerator aur denominator ko $e^{r_{θ} (x, y_{w})}$ se divide karo:

$P = \frac{1}{1 + e ^{- (r_{θ} (x, y_{w}) - r_{θ} (x, y_{l}))}} = σ (r_{θ} (x, y_{w}) - r_{θ} (x, y_{l}))$

Yeh step kyun? Yeh dikhata hai ki model sirf rewards ke difference ki parwah karta hai — exactly jo ek comparison hume deta hai. Absolute scale unidentified hai (sabhi rewards mein ek constant add karna kuch nahi badlata), jo theek hai.

Ab observed human choices ki likelihood maximize karo → negative log-likelihood minimize karo:

RM usually SFT model hota hai jiska final token-prediction head ek single scalar head se replace kar diya jaata hai last token par.

Stage 3 — PPO ke saath RL optimization

KYA: LM ko ek policy $π_{ϕ} (y ∣ x)$ maano. Ek response sample karo, use RM se score karo, aur $ϕ$ ko expected reward badhane ke liye update karo.

Naive objective: $max_{ϕ} E_{x, y \sim π_{ϕ}} [r_{θ} (x, y)]$

YEH DANGEROUS KYUN HAI: RM sirf us data ke paas accurate hai jo usne dekha. Agar hum pure reward par hard push karein, policy adversarial gibberish dhundh leti hai jo RM ko fool kare (reward hacking). Isliye hum SFT policy ki taraf ek leash add karte hain.

KL term KYUN (intuition ki derivation): Hum chahte hain high reward lekin ek trusted model ke paas rehna bhi. Per-token reward ko $r^{'} (x, y) = r_{θ} (x, y) - β lo g \frac{π _{ϕ}}{π ^{SFT}}$ se rewrite karo. Agar $π_{ϕ}$ $π^{SFT}$ se bahut door jata hai, to log-ratio blow up ho jaata hai aur reward kha jaata hai. Yeh generations ko fluent rakhta hai aur mode collapse prevent karta hai.

PPO policy ko kaise update karta hai? PPO reward maximize karta hai jabki har update kitna door policy ko move karta hai usse limit karta hai, clipped surrogate objective use karke. Ratio $ρ_{t} = \frac{π _{ϕ} ( a _{t} ∣ s _{t} )}{π _{ϕ_{old}} ( a _{t} ∣ s _{t} )}$ aur advantage $A_{t}$ ke saath:

$L^{P P O} = E [min (ρ_{t} A_{t}, clip (ρ_{t}, 1 - ϵ, 1 + ϵ) A_{t})]$

Clip kyun? Ek noisy reward par ek bada single update policy ko barbad kar sakta hai. Clipping $ρ_{t}$ ko $[1 - ϵ, 1 + ϵ]$ se aage move karne ka incentive remove karta hai, chhhote, safe steps deta hai.

Worked Example 1 — Reward-model loss numerically

Maano ek comparison ke liye $r_{θ} (x, y_{w}) = 2.0$ aur $r_{θ} (x, y_{l}) = 0.5$ hai.

Step: difference $Δ = 2.0 - 0.5 = 1.5$ . Kyun? RM sirf gap dekhta hai.
Step: $σ (1.5) = \frac{1}{1 + e ^{- 1.5}} \approx 0.8176$ . Kyun? Probability jo humne correctly winner predict ki.
Step: loss $= - lo g (0.8176) \approx 0.201$ . Kyun? Kam loss kyunki model ne already winner ko higher rank diya; gradient phir bhi dheere se gap widen karega.

Agar model galat tha ( $r_{w} = 0.5, r_{l} = 2.0$ ): $σ (- 1.5) \approx 0.182$ , loss $= - lo g (0.182) \approx 1.70$ — bahut bada, ek strong correction push karta hai. Yahi to hum chahte hain.

Worked Example 2 — KL leash in action

Maano ek token ke liye SFT model probability $0.10$ deta hai ek word ko aur current policy $0.40$ deta hai.

Log-ratio $= lo g (0.40/0.10) = lo g 4 \approx 1.386$ .
$β = 0.2$ ke saath, penalty $= 0.2 \times 1.386 \approx 0.277$ reward se subtract hoti hai.
Yeh step kyun? Policy is token par trusted model se 4× zyada confident ho gayi; penalty kehti hai "aisa tabhi karo jab reward gain clearly $0.277$ se zyada ho."

Common mistakes

Recall Feynman: ek 12-saal ke bachche ko explain karo

Socho tum ek puppy ko tricks sikhaa rahe ho. Tum English mein explain nahi kar sakte ki "achha behavior" kya hota hai, lekin tum do tricks point karke keh sakte ho "woh behtar tha." Tum ek treat-predicting judge (reward model) ko bahut saari yeh comparisons dete ho jab tak woh guess karna na seekh le ki tumhein kaunsi trick pasand aayegi. Phir tum puppy ko cheezein try karne do, judge pretend-treats deta hai, aur puppy seekhta hai ki zyada treats kya kamaata hai. Lekin tum puppy ko ek leash (KL penalty) par rakhte ho taaki woh judge ko fool karne ke liye weird cheezein karne na bhaage. Wohi leashed treat-training RLHF hai.

Active-recall flashcards

#flashcards/ai-ml

RLHF ke 3 stages kya hain, order mein?

(1) Supervised Fine-Tuning (SFT), (2) human preference comparisons se ek Reward Model train karna, (3) PPO + KL penalty ke saath RL optimization.

Absolute quality scores ki jagah pairwise comparisons kyun use karte hain?

Humans noisy absolute ratings dete hain lekin reliable relative judgments dete hain ("A > B"); comparisons saste aur zyada consistent hote hain.

Reward-model loss likho aur uska origin batao.

- E [lo g σ (r_{θ} (x, y_{w}) - r_{θ} (x, y_{l}))]

, Bradley–Terry model + maximum likelihood se derived.

RM sirf rewards ke difference par kyun depend karta hai?

Bradley–Terry probability

σ (r_{w} - r_{l})

mein simplify hoti hai; sabhi rewards mein ek constant add karna loss unchanged chodta hai (scale unidentified hai).

Full RLHF RL objective kya hai?

max_{ϕ} E_{y \sim π_{ϕ}} [r_{θ} (x, y) - β lo g \frac{π _{ϕ} ( y ∣ x )}{π ^{S F T} ( y ∣ x )}]

— reward minus SFT policy ke liye KL penalty.

KL penalty kyun include karte hain?

Policy ko trusted SFT model ke paas rakhne ke liye, fluency/diversity preserve karne aur reward hacking / mode collapse prevent karne ke liye.

RLHF mein reward hacking (Goodhart) kya hai?

Policy proxy reward model ki inaccuracies exploit karti hai, aisi outputs produce karti hai jo high score karti hain lekin actually buri hoti hain.

PPO ka clipping kya accomplish karta hai?

Yeh limit karta hai ki har update policy ko kitna door move kare (ratio 1 ke paas rakha jaata hai), chhhote stable steps deta hai aur noisy rewards se destructive updates avoid karta hai.

Kya RLHF nayi knowledge add karta hai?

Mostly nahi — yeh un behaviors ko reshape/prioritize karta hai jo base model mein pehle se thi; yeh nayi facts nahi sikhata.

Reward model ki architecture usually kya hoti hai?

SFT model jiska token head ek single scalar reward head se replace kar diya jaata hai final token par.

Connections

Supervised Fine-Tuning (SFT) — starting policy aur reference model provide karta hai.
Proximal Policy Optimization (PPO) — RL algorithm jo update karta hai.
KL Divergence — woh leash jo policy ko SFT ke paas rakhti hai.
Bradley-Terry Model — reward loss ka statistical basis.
Direct Preference Optimization (DPO) — explicit RM/RL loop ke bina RLHF.
Reward Hacking and Goodhart's Law — woh failure jisse RLHF ko bachna hai.
Constitutional AI / RLAIF — human labels ki jagah AI feedback lena.
Prompting aur Retrieval-Augmented Generation (RAG) — alternative/complementary alignment levers.