Policy gradient — REINFORCE

5.6.18 · Coding › Machine Learning (Aerospace Applications)

REINFORCE KYA hai?

Policy gradients kyun use karein? Value-based methods (Q-learning) mein aap values seekhte ho aur greedily act karte ho. Lekin continuous ya high-dimensional action spaces ke liye (jaise ek aircraft ke control-surface deflections), actions par maximize karna bahut mushkil ho jaata hai. Policy methods seedha action distribution ko parameterize karte hain aur $\theta$ ko gradient ascent se optimize karte hain — koi argmax ki zaroorat nahi.

KAISE: gradient scratch se derive karna

Hum chahte hain ${\nabla }_{\theta }J(\theta )$. Mushkil yeh hai: expectation $\tau$ par hai, jiski distribution $\theta$ par depend karti hai. Hum gradient ko seedha expectation ke andar naively push nahi kar sakte.

Step 1 — expectation ko integral ke roop mein likhna. $p_{\theta }(\tau )$ ko trajectory $\tau$ ki probability maano: $J(\theta )=\int p_{\theta }(\tau )R(\tau )d\tau .$ Yeh step kyun? Ek expectation hoti hi hai probability-weighted integral; isse explicitly likhne se hum differentiate kar sakte hain.

Step 2 — differentiate karo. Sirf $p_{\theta }$ hi $\theta$ par depend karta hai: ${\nabla }_{\theta }J=\int {\nabla }_{\theta }{p}_{\theta }(\tau )R(\tau )d\tau .$

Step 3 — log-derivative trick. Hum yeh identity use karte hain: ${\nabla }_{\theta }{p}_{\theta }(\tau )=p_{\theta }(\tau ){\nabla }_{\theta }\log p_{\theta }(\tau ),$ jo simply $\nabla \log f=\frac{\nabla f}{f}$ ko rearrange karna hai. Yeh step kyun? Yeh $p_{\theta }(\tau )$ ko weight ki tarah wapas insert karta hai, integral ko ek sampleable expectation mein badal deta hai: ${\nabla }_{\theta }J=\int p_{\theta }(\tau ){\nabla }_{\theta }\log p_{\theta }(\tau )R(\tau )d\tau ={\mathbb{E}}_{\tau }[{\nabla }_{\theta }\log p_{\theta }(\tau )R(\tau )].$

Step 4 — $\log p_{\theta }(\tau )$ expand karo. Trajectory probability factorize hoti hai: $p_{\theta }(\tau )=\underset{\text{env}}{\underbrace{p(s_0)}}{\prod }_{t=0}^T\underset{\text{policy}}{\underbrace{{\pi }_{\theta }(a_t\mid s_t)}}\underset{\text{env dynamics}}{\underbrace{p(s_{t+1}\mid s_t,a_t)}}.$ $\log$ lene se products sums ban jaate hain; phir ${\nabla }_{\theta }$ har us term ko khatam kar deta hai jisme $\theta$ nahi hai (environment terms $p(s_0)$ aur $p(s_{t+1}\mid s_t,a_t)$): ${\nabla }_{\theta }\log p_{\theta }(\tau )={\sum }_{t=0}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t).$ Yeh kyun matter karta hai: hume environment ka koi model nahi chahiye. Yahi REINFORCE ka jaadu hai.

Reward-to-go aur baselines (woh 20% jo 80% kaam deta hai)

Reward-to-go. Time $t$ par liya gaya action un rewards ko affect nahi kar sakta jo pehle ho chuke hain. Toh poore $R(\tau )$ ki jagah $t$ ke baad ka return use karo: ${\nabla }_{\theta }J=\mathbb{E}\left[{\sum }_t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)\underset{G_t}{\underbrace{\sum _{t^{\prime }\ge t}{\gamma }^{t^{\prime }-t}{r}_{t^{\prime }}}}\right].$ Kyun? Action se independent irrelevant reward hatane se variance kam hota hai bina koi bias add kiye.

Baseline. Koi bhi aisi function $b(s_t)$ subtract karo jo $a_t$ par depend na kare: ${\nabla }_{\theta }J=\mathbb{E}\left[{\sum }_t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)(G_t-b(s_t))\right].$ Yeh unbiased hai kyunki ${\mathbb{E}}_a[{\nabla }_{\theta }\log {\pi }_{\theta }(a\mid s)b(s)]=b(s){\nabla }_{\theta }\underset{=1}{\underbrace{\sum _a{\pi }_{\theta }(a\mid s)}}=0.$ Ek achha baseline $b(s)=V(s)$ hai; tab $G_t-b(s_t)$ ek advantage estimate ban jaata hai.

Worked examples

Common mistakes (steel-manned)

Active recall

Connections

• Value-based methods (Q-learning) — woh alternative jo policies nahi, values seekhti hai.
• Actor-Critic methods — ek learned $V(s)$ ko baseline/critic ki tarah use karta hai.
• Advantage function A(s,a) — jo $G_t-b(s_t)$ estimate karta hai.
• Softmax and log-softmax gradients — discrete policies ke liye reuse hota hai.
• Gaussian policies for continuous control — aerospace mein thrust/deflection commands.
• Monte Carlo estimation and variance reduction — baselines kyun help karte hain.
• Gradient ascent / stochastic optimization — update engine.

Concept Map