Policy gradient methods — Hinglish notes

5.2.5 · AI-ML › Deep & Advanced RL

WHY: hum policy gradients chahte hi kyun hain?

Value methods (Q-learning) $Q (s, a)$ seekhte hain aur $ar g max_{a} Q$ pick karte hain. Yeh tab fail hota hai jab:

Actions continuous hoon — ek infinite set par argmax karna aasaan nahi hota.
Optimal policy stochastic ho — jaise rock-paper-scissors, ya partially observed states jahan best move genuinely random ho. Ek greedy value policy deterministic hoti hai.
Aap smooth improvement chahte ho — chhote $θ$ changes se chhote policy changes aate hain, jo stable hota hai.

WHAT: hum optimize kya kar rahe hain?

Objective define karo as ek trajectory $τ = (s_{0}, a_{0}, s_{1}, a_{1}, \dots)$ ka expected return:

$J (θ) = E_{τ \sim π_{θ}} [R (τ)], R (τ) = \sum_{t = 0}^{T} γ^{t} r_{t}$

Hum gradient ascent karte hain: $θ \leftarrow θ + α \nabla_{θ} J (θ)$ .

Pura game yeh hai: hum $\nabla_{θ} J$ compute kaise karein jab reward obviously $θ$ par depend nahi karta? Reward environment se aata hai; $θ$ sirf shape karta hai ki hum kaun si trajectories sample karte hain. Yahi trick hai.

HOW: Policy Gradient Theorem scratch se derive karna

Objective ko trajectories par ek integral ki tarah likho, jahan $P_{θ} (τ)$ trajectory probability hai:

$J (θ) = \int P_{θ} (τ) R (τ) d τ$

Step 1 — differentiate karo. $\nabla_{θ} J = \int \nabla_{θ} P_{θ} (τ) R (τ) d τ$ Yeh step kyun? $R (τ)$ mein koi $θ$ nahi hai (reward environment ka hai), isliye sirf $P_{θ}$ differentiate hota hai.

Step 2 — log-derivative trick. $P_{θ} (τ)$ se multiply aur divide karo: $\nabla_{θ} P_{θ} = P_{θ} \frac{\nabla _{θ} P _{θ}}{P _{θ}} = P_{θ} \nabla_{θ} lo g P_{θ} (τ)$ Kyun? Kyunki $\nabla lo g f = \frac{\nabla f}{f}$ . Yeh crucial identity hai — yeh ek probability ke gradient ko ek expectation mein convert kar deti hai jise hum sample kar sakte hain.

$\nabla_{θ} J = \int P_{θ} (τ) \nabla_{θ} lo g P_{θ} (τ) R (τ) d τ = E_{τ} [\nabla_{θ} lo g P_{θ} (τ) R (τ)]$

Step 3 — $lo g P_{θ} (τ)$ expand karo. Trajectory probability factorize hoti hai: $P_{θ} (τ) = init ρ (s_{0}) \prod_{t} dynamics P (s_{t + 1} ∣ s_{t}, a_{t}) policy π_{θ} (a_{t} ∣ s_{t})$ Log lo → sum ban jaata hai. Init aur dynamics terms mein koi $θ$ nahi hai, isliye $\nabla_{θ}$ ke under woh vanish ho jaate hain: $\nabla_{θ} lo g P_{θ} (τ) = \sum_{t = 0}^{T} \nabla_{θ} lo g π_{θ} (a_{t} ∣ s_{t})$ Yeh kyun matter karta hai: humein environment ka dynamics model kabhi nahi chahiye! Yahi model-free hai.

Variance reduce karna: causality + baseline

REINFORCE kaam karta hai lekin iska variance bahut bada hota hai. Do principled fixes hain:

1. Causality (reward-to-go). Time $t$ par ek action $t$ se pehle ke rewards ko affect nahi kar sakta. Isliye $t$ -wein term ko multiply karne wale poore $R (τ)$ ki jagah sirf future reward use karo: $\nabla_{θ} J = E \sum_{t} \nabla_{θ} lo g π_{θ} (a_{t} ∣ s_{t}) G_{t} (reward-to-go) t^{'} \geq t \sum γ^{t^{'} - t} r_{t^{'}}$

2. Baseline subtraction. Koi bhi function $b (s_{t})$ subtract karo jo action par depend na kare: $\nabla_{θ} J = E [\sum_{t} \nabla_{θ} lo g π_{θ} (a_{t} ∣ s_{t}) (G_{t} - b (s_{t}))]$ Yeh allowed kyun hai (unbiased)? Kyunki $E_{a \sim π} [\nabla_{θ} lo g π_{θ} (a ∣ s) b (s)] = b (s) \nabla_{θ} \sum_{a} π_{θ} (a ∣ s) = b (s) \nabla_{θ} 1 = 0$ . Zero subtract karna expectation mein kuch nahi badalta lekin variance kaafi kam kar deta hai.

Sabse acchi baseline $b (s) = V (s)$ hai, jo advantage $A (s, a) = G_{t} - V (s) \approx Q (s, a) - V (s)$ deti hai. Yahi Actor-Critic ka seed hai.

Steel-manned mistakes

Flashcards

Policy gradients mein log-derivative trick kaun si core problem solve karta hai?

Yeh

\nabla_{θ} P_{θ} (τ)

P_{θ} (τ) \nabla_{θ} lo g P_{θ} (τ)

mein convert karta hai, ek probability ke gradient ko ek samplable expectation mein badal deta hai.

Environment dynamics policy gradient ke liye irrelevant kyun hain?

Unmein koi

θ

nahi hota, isliye

\nabla_{θ} lo g P (s^{'} ∣ s, a) = 0

; sirf

\nabla_{θ} lo g π_{θ}

survive karta hai — PG ko model-free banata hai.

REINFORCE gradient state karo.

\nabla_{θ} J = E_{τ} [\sum_{t} \nabla_{θ} lo g π_{θ} (a_{t} ∣ s_{t}) R (τ)]

Policy gradients Q-learning se continuous actions better kyun handle kar sakti hain?

Yeh directly policy ko parameterize karti hain aur actions sample karti hain, ek infinite action set par

ar g max

se bachti hain.

Ek baseline

b (s)

subtract karna unbiased kyun hai?

Kyunki

E_{a \sim π} [\nabla_{θ} lo g π_{θ} (a ∣ s) b (s)] = b (s) \nabla_{θ} \sum_{a} π = b (s) \nabla_{θ} 1 = 0

Advantage function kya hai aur

V (s)

acchi baseline kyun hai?

A (s, a) = Q (s, a) - V (s)

;

b = V

use karna measure karta hai ki ek action average se kitna better hai, variance minimize karta hai.

Har term ke liye full return ki jagah "reward-to-go" kyun?

Causality: time

t

par ek action

t

se pehle ke rewards ko influence nahi kar sakta, isliye woh terms irrelevant noise hain aur drop kar diye jaate hain.

J (θ)

ke liye ascent ya descent?

Gradient ascent —

J

ek return hai jise hum maximize karte hain.

Recall Feynman: ek 12-saal ke bacche ko samjhao

Socho ek kutte ko treats se train kar rahe ho. Tum kutte ko exactly kaise apni taangein hilaane hain yeh nahi batate. Tum bas woh moves jo treats mile zyada hone dete ho aur buri moves kam hone dete ho. Dog policy hai; treat reward hai. Policy gradients = "kya us trajectory ko bada treat mila? Toh usme jo bhi choices ki hain unhe thoda zyada likely banao." Math (log-derivative trick) sirf yeh careful bookkeeping hai ki "main dice ko kaise nudge karun taaki acche rolls zyada aayein?"

Connections

Actor-Critic methods — baseline ek learned critic $V_{ϕ}$ ban jaati hai.
REINFORCE algorithm — direct Monte-Carlo estimator.
Advantage function aur Generalized Advantage Estimation.
Trust Region and PPO — stability ke liye step size control karo.
Value-based methods (Q-learning) — woh alternative family jisse PG contrast karta hai.
Log-derivative trick / Score function estimator — ML mein same idea.
Variance reduction in Monte Carlo.