Reward shaping and sparse rewards

5.2.13 · AI-ML › Deep & Advanced RL

Sparse reward KYA hota hai?

YEH mushkil kyun hai: RL return $G_t={\sum }_{k\ge 0}{\gamma }^k{r}_{t+k}$ se seekhta hai. Agar almost sab $r=0$ hain, toh $Q(s,a)\approx 0$ har jagah jahan agent actually gaya ho, isliye koi bhi action dusre se behtar nahi lagta → koi policy improvement nahi jab tak goal luck se na mile. Luck se goal milne ki probability steps ki sankhya ke saath exponentially decay hoti hai.

Reward shaping KYA hai?

Danger yeh hai: ek naïve $F$ reward hacking create kar sakta hai — agent ek aisa loop dhundh leta hai jo $F$ ko milkify kare bina task solve kiye.

Safe tarike se shape kaise karein: Potential-Based Reward Shaping (PBRS)

First principles se derivation — Optimal policy kyun preserve hoti hai

Hum ise derive karenge, memorize nahi karenge.

Step 1 — Shaped return likho. Ek trajectory $s_0,s_1,s_2,\dots$ ke liye shaped return hai $G_0^{\prime }={\sum }_{t=0}^{\infty }{\gamma }^t(r_t+\gamma \Phi (s_{t+1})-\Phi (s_t)).$ Yeh step kyun? By definition return shaped reward $r_t^{\prime }=r_t+F_t$ ka discounted sum hai.

Step 2 — Sum ko split karo. $G_0^{\prime }=\underset{G_0}{\underbrace{\sum _t{\gamma }^t{r}_t}}+{\sum }_{t=0}^{\infty }{\gamma }^t(\gamma \Phi (s_{t+1})-\Phi (s_t)).$ Kyun? Sum of a sum linearly split hota hai.

Step 3 — Shaping part ko telescope karo. Maano $T={\sum }_t{\gamma }^t(\gamma \Phi (s_{t+1})-\Phi (s_t))$. Ise likhte hain: $T=(\gamma \Phi (s_1)-\Phi (s_0))+\gamma (\gamma \Phi (s_2)-\Phi (s_1))+{\gamma }^2(\gamma \Phi (s_3)-\Phi (s_2))+\cdots$ $\Phi (s_1)$ wale terms group karo: coefficient $=\gamma -\gamma \cdot 1=0$. Similarly $\Phi (s_2)$ ke liye: ${\gamma }^2-{\gamma }^2=0$. Sab kuch cancel ho jaata hai sirf $-\Phi (s_0)$ chhodkar. Kyun? Har $\Phi (s_t)$ ek term se $+\gamma \cdot {\gamma }^{t-1}={\gamma }^t$ ke saath appear hota hai aur agle term se $-{\gamma }^t$ ke saath — yeh annihilate ho jaate hain. Yeh ek telescoping series hai.

Step 4 — Collect karo. $G_0^{\prime }=G_0-\Phi (s_0).$

Step 5 — Interpret karo. Shaped value true value se ek aisi constant se different hai jo sirf start state pe depend karti hai, actions pe nahi: $V^{\prime \pi }(s)=V^{\pi }(s)-\Phi (s),Q^{\prime \pi }(s,a)=Q^{\pi }(s,a)-\Phi (s).$ Yeh punchline kyun hai: $\arg {\max }_a{Q}^{\prime }(s,a)=\arg {\max }_a(Q(s,a)-\Phi (s))=\arg {\max }_{a}Q(s,a)$ kyunki $\Phi (s)$ ek constant offset over $a$ hai. Toh greedy/optimal policy identical rehti hai. ∎

Sparse rewards ke liye aur tools (80/20)

• Intrinsic motivation / curiosity: $F=\eta \Vert \text{prediction error}\Vert$ ya count bonus $F=\beta /\sqrt{N(s)}$ add karo taaki agent ko novel states visit karne ka reward mile → explore karta rehta hai jab tak real reward na mile.
• Hindsight Experience Replay (HER): agar agent goal $g$ ke liye gaya tha lekin $g^{\prime }$ tak pahuncha, toh episode ko relabel karo jaise $g^{\prime }$ hi goal tha. Ab ek "failure" ek "success" ban jaata hai → dense synthetic learning signal, koi reward hacking nahi.
• Curriculum learning: agent ke paas wale goals se shuru karo (dense success), phir dheere dheere unhe door karte jao.

Worked examples

Common mistakes (unhe Steel-man karo)

Flashcards

Connections

• Markov Decision Processes — shaping $R$ modify karta hai lekin ${\pi }^{\ast }$ preserve karna zaroori hai.
• Value Functions and Bellman Equations — $V^{\prime \pi }=V^{\pi }-\Phi$ derivation.
• Exploration vs Exploitation — curiosity bonuses as intrinsic reward.
• Hindsight Experience Replay — sparse-reward remedy without hacking.
• Curriculum Learning — goals order karke sparsity ko ease karna.
• Reward Hacking and Specification Gaming — failure mode jo PBRS prevent karta hai.

Concept Map