Direct Preference Optimization (DPO)

4.4.4 · AI-ML › Alignment, Prompting & RAG

DPO exist kyun karta hai?

Setup (pieces KYA hain)

First principles se DERIVATION

Hum DPO ko teen moves mein derive karte hain. Har "Why this step?" ko follow karo.

Step 1 — RLHF objective

RLHF expected reward maximize karta hai, reference model ke paas rehte hue:

${\max }_{{\pi }_{\theta }}{\mathbb{E}}_{x,y\sim {\pi }_{\theta }}[r(x,y)]-\beta \mathrm{KL}({\pi }_{\theta }(y|x)\Vert {\pi }_{\text{ref}}(y|x))$

Why this step? Reward term high-quality answers ki taraf push karta hai; KL term ek leash ki tarah hai jo model ko gibberish produce karne se rokta hai jo reward ko game kare (reward hacking).

Step 2 — OPTIMAL policy solve karo

Is objective ka ek known closed-form optimum hai (KL-regularized reward maximization ka ek standard result):

${\pi }^{\ast }(y|x)=\frac{1}{Z(x)}{\pi }_{\text{ref}}(y|x)\exp (\frac{1}{\beta }r(x,y))$

jahan $Z(x)={\sum }_y{\pi }_{\text{ref}}(y|x)\exp (\frac{1}{\beta }r(x,y))$ partition function (normalizer) hai.

Why this step? "Given reward, best policy" ko "given policy, implied reward" mein turn karne ke liye. $Z(x)$ intractable hai (saari sequences pe sum karta hai) — yahi wo villain hai jise hume eliminate karna hai.

Step 3 — Invert karo taaki policy ke zariye reward express ho sake

Step 2 ke logs lo aur $r$ ke liye solve karo:

$r(x,y)=\beta \log \frac{{\pi }^{\ast }(y|x)}{{\pi }_{\text{ref}}(y|x)}+\beta \log Z(x)$

Why this step? Ab reward policy ke terms mein likha gaya hai. Messy $Z(x)$ term sirf $x$ pe depend karta hai, $y$ pe nahi — yeh yaad rakho.

Step 4 — Bradley–Terry preference model mein plug karo

Human preferences ko Bradley–Terry se model kiya jaata hai: probability ki $y_w$, $y_l$ ko beat karta hai, ye hai:

$P(y_w\succ y_l\mid x)=\sigma (r(x,y_w)-r(x,y_l))$

jahan $\sigma$ sigmoid hai. Step 3 ka reward substitute karo:

$r(x,y_w)-r(x,y_l)=\beta \log \frac{{\pi }^{\ast }(y_w|x)}{{\pi }_{\text{ref}}(y_w|x)}-\beta \log \frac{{\pi }^{\ast }(y_l|x)}{{\pi }_{\text{ref}}(y_l|x)}+\underset{=0}{\underbrace{\beta \log Z(x)-\beta \log Z(x)}}$

Why this step? Yahi to magic hai. Kyunki hum same prompt $x$ ke liye rewards ka difference lete hain, intractable $\log Z(x)$ cancel ho jaata hai. Na reward model, na partition function.

Step 5 — Loss likho

${\pi }^{\ast }$ ko apne trainable ${\pi }_{\theta }$ se replace karo aur observed preferences ka log-likelihood maximize karo (equivalently minimize karo):

Gradient kaisa behave karta hai? (intuition check)

Gradient hai:

${\nabla }_{\theta }\mathcal{L}=-\beta \mathbb{E}\left[\underset{\text{weight}}{\underbrace{\sigma ({\hat{r}}_l-{\hat{r}}_w)}}\right({\nabla }_{\theta }\log {\pi }_{\theta }(y_w|x)-{\nabla }_{\theta }\log {\pi }_{\theta }(y_l|x)\left)\right]$

Worked examples

Common mistakes (Steel-manned)

Active recall

Connections

• RLHF - Reinforcement Learning from Human Feedback — parent method jise DPO simplify karta hai.
• PPO - Proximal Policy Optimization — wo RL algorithm jise DPO remove karta hai.
• Bradley-Terry Model — Step 4 mein use hone wala preference likelihood.
• KL Divergence — regularizer jiska closed-form optimum derivation shuru karta hai.
• Reward Modeling — DPO mein implicit ho jaata hai.
• Supervised Fine-Tuning (SFT) — dono ${\pi }_{\theta }$ init aur ${\pi }_{\text{ref}}$ provide karta hai.
• Sigmoid and Logistic Loss — DPO loss bas reward margins pe logistic regression hai.

Concept Map