Policies and value functions — Hinglish notes

5.1.3 · AI-ML › Reinforcement Learning Foundations

What Is a Policy?

Stochastic kyun? Kuch environments mein (partially observable, ya mixed strategies wale games mein), actions ko randomize karna optimal hota hai. Yeh learning ke dauran exploration mein bhi help karta hai.

Yeh kaise kaam karta hai?

Agent state $s_{t}$ observe karta hai
Policy action output karti hai: $a_{t} \sim π (\cdot ∣ s_{t})$ (distribution se sample karo)
Environment reward $r_{t + 1}$ aur next state $s_{t + 1}$ ke saath respond karta hai
Repeat karo

State-Value Function: V(s)

First principles se derivation:

"Expected future return" ki definition se start karo jab tum time $t$ par state $s$ mein ho.

Return (cumulative reward) hai: $G_{t} = R_{t + 1} + γ R_{t + 2} + γ^{2} R_{t + 3} + \dots = \sum_{k = 0}^{\infty} γ^{k} R_{t + k + 1}$

Discount kyun?

Abhi ke rewards zyada certain hain baad ke rewards se (uncertainty)
Mathematically convergence ensure karta hai jab rewards bounded hon
Impatience model karta hai (economic interpretation)

Policy $π$ ke under state $s$ ki value expected return hai: $V^{π} (s) = E_{π} [G_{t} ∣ S_{t} = s]$

Expectation kyun? Policy stochastic ho sakti hai, aur environment dynamics $P (s^{'} ∣ s, a)$ bhi aksar stochastic hoti hain, isliye hum saare possible futures par average karte hain.

$V^{π}$ ke liye Bellman Equation:

Hum return ko recursively decompose kar sakte hain: $G_{t} = R_{t + 1} + γ G_{t + 1}$

Dono sides par expectation lete hain: $V^{π} (s) = E_{π} [R_{t + 1} + γ G_{t + 1} ∣ S_{t} = s]$

Ab expectation split karo. Pehle policy se action sample karo, phir dynamics se next state: $V^{π} (s) = \sum_{a} π (a ∣ s) \sum_{s^{'}} P (s^{'} ∣ s, a) [R (s, a, s^{'}) + γ V^{π} (s^{'})]$

Yeh step kyun? Hum saare possible actions par marginalize karte hain (policy ke weight se) aur saare possible next states par (transition probability ke weight se).

Compact form: $V^{π} (s) = a \sum π (a ∣ s) s^{'}, r \sum p (s^{'}, r ∣ s, a) [r + γ V^{π} (s^{'})]$

Yahi $V^{π}$ ke liye Bellman expectation equation hai.

Worked example V(s) Hand se Compute karna

Setup: 2-state MDP

States: $S = {s_{1}, s_{2}}$
Actions: $A = {a_{1}}$ (sirf ek action)
Policy: $π (a_{1} ∣ s_{1}) = 1$ , $π (a_{1} ∣ s_{2}) = 1$ (deterministic)
Dynamics:
$s_{1}$ se: $a_{1}$ lo → reward $+ 1$ milta hai, prob 0.5 se $s_{1}$ mein raho, prob 0.5 se $s_{2}$ jao
$s_{2}$ se: $a_{1}$ lo → reward $+ 2$ milta hai, hamesha ke liye $s_{2}$ mein raho
Discount: $γ = 0.9$

Dhundho: $V^{π} (s_{1})$ aur $V^{π} (s_{2})$

Solution: Bellman equations likho: $V^{π} (s_{1}) = 1 + 0.9 [0.5 \cdot V^{π} (s_{1}) + 0.5 \cdot V^{π} (s_{2})]$ $V^{π} (s_{2}) = 2 + 0.9 \cdot V^{π} (s_{2})$

Yeh form kyun?

Immediate reward yahaan deterministic hai (hamesha $s_{1}$ se +1, $s_{2}$ se +2)
Future value next state ki discounted expected value hai Pehle doosri equation solve karo: $V^{π} (s_{2}) = 2 + 0.9 V^{π} (s_{2})$ $V^{π} (s_{2}) - 0.9 V^{π} (s_{2}) = 2$ $0.1 V^{π} (s_{2}) = 2$ $V^{π} (s_{2}) = 20$

Kyun? $s_{2}$ mein agent har step hamesha +2 pata hai, toh discounted sum hai $2 + 0.9 \cdot 2 + 0. 9^{2} \cdot 2 + \dots = \frac{2}{1 - 0.9} = 20$ .

Pehli equation mein plug in karo: $V^{π} (s_{1}) = 1 + 0.9 [0.5 V^{π} (s_{1}) + 0.5 \cdot 20]$ $V^{π} (s_{1}) = 1 + 0.45 V^{π} (s_{1}) + 9$ $V^{π} (s_{1}) - 0.45 V^{π} (s_{1}) = 10$ $0.55 V^{π} (s_{1}) = 10$ $V^{π} (s_{1}) = \frac{10}{0.55} \approx 18.18$

Interpretation: $s_{1}$ thoda $s_{2}$ se worse hai kyunki tumhe immediately reward sirf +1 milta hai, phir shayad $s_{1}$ mein hi raho (lower reward ko aur prolonge karo) jab tak eventually $s_{2}$ na pahuncho.

Action-Value Function: Q(s,a)

Q vs V kyun?

$V^{π} (s)$ batata hai "state $s$ kitni achi hai"
$Q^{π} (s, a)$ batata hai "state $s$ mein action $a$ lena kitna acha hai"
Q zyada granular hai—tum different actions ke Q-values compare karke policy improve kar sakte ho.

Q aur V ke beech relationship:

$V^{π} (s) = \sum_{a} π (a ∣ s) Q^{π} (s, a)$

Kyun? State ki value policy se sample kiye gaye actions par expected Q-value hoti hai.

$Q^{π}$ ke liye Bellman Equation:

First principles se derive karo. State $s$ mein action $a$ lene ke baad ka return: $G_{t} = R_{t + 1} + γ G_{t + 1}$

Toh: $Q^{π} (s, a) = E_{π} [R_{t + 1} + γ G_{t + 1} ∣ S_{t} = s, A_{t} = a]$

$a$ lene ke baad, tum $s^{'}$ mein transition karte ho, phir wahaan se policy $π$ follow karte ho: $Q^{π} (s, a) = \sum_{s^{'}, r} p (s^{'}, r ∣ s, a) [r + γ \sum_{a^{'}} π (a^{'} ∣ s^{'}) Q^{π} (s^{'}, a^{'})]$

Kyun?

Immediate reward $r$ transition $(s, a) \to s^{'}$ se aata hai
Future value $\sum_{a^{'}} π (a^{'} ∣ s^{'}) Q^{π} (s^{'}, a^{'})$ bas $V^{π} (s^{'})$ hai

Compact form: $Q^{π} (s, a) = s^{'}, r \sum p (s^{'}, r ∣ s, a) [r + γ V^{π} (s^{'})]$

Ya equivalently: $Q^{π} (s, a) = s^{'}, r \sum p (s^{'}, r ∣ s, a) [r + γ a^{'} \sum π (a^{'} ∣ s^{'}) Q^{π} (s^{'}, a^{'})]$

Worked example Q-Function Calculation

Setup: Same 2-state MDP, lekin ab $s_{1}$ se do actions hain:

$s_{1}$ se:
Action $a_{1}$ : reward +1, deterministically $s_{2}$ jao
Action $a_{2}$ : reward +0.5, deterministically $s_{1}$ mein raho
$s_{2}$ se: sirf $a_{1}$ available, reward +2, $s_{2}$ mein raho
Policy: $π (a_{1} ∣ s_{1}) = 0.8$ , $π (a_{2} ∣ s_{1}) = 0.2$
Discount: $γ = 0.9$

Dhundho: $Q^{π} (s_{1}, a_{1})$ aur $Q^{π} (s_{1}, a_{2})$ Solution: Pehle, $V^{π} (s_{2}) = 20$ (pehle compute kiya tha).

$Q^{π} (s_{1}, a_{1})$ ke liye: $Q^{π} (s_{1}, a_{1}) = + 0.9 \cdot V^{π} (s_{2}) = 1 + 0.9 \cdot 20 = 1 + 18 = 19$

Kyun? $a_{1}$ lene se immediate reward +1 milta hai, phir tum $s_{2}$ mein ho jiska value 20 hai.

$Q^{π} (s_{1}, a_{2})$ ke liye: $Q^{π} (s_{1}, a_{2}) = 0.5 + 0.9 \cdot V^{π} (s_{1})$ Lekin hamein $V^{π} (s_{1})$ chahiye: $V^{π} (s_{1}) = 0.8 \cdot Q^{π} (s_{1}, a_{1}) + 0.2 \cdot Q^{π} (s_{1}, a_{2})$ $V^{π} (s_{1}) = 0.8 \cdot 19 + 0.2 \cdot Q^{π} (s_{1}, a_{2})$ $V^{π} (s_{1}) = 15.2 + 0.2 \cdot Q^{π} (s_{1}, a_{2})$ Aur: $Q^{π} (s_{1}, a_{2}) = 0.5 + 0.9 V^{π} (s_{1})$ Substitute karo: $Q^{π} (s_{1}, a_{2}) = 0.5 + 0.9 (15.2 + 0.2 Q^{π} (s_{1}, a_{2}))$ $Q^{π} (s_{1}, a_{2}) = 0.5 + 13.68 + 0.18 Q^{π} (s_{1}, a_{2})$ $0.82 Q^{π} (s_{1}, a_{2}) = 14.18$ $Q^{π} (s_{1}, a_{2}) \approx 17.29$

Interpretation: Action $a_{1}$ (Q=19) state $s_{1}$ mein $a_{2}$ (Q≈17.29) se better hai. Policy already $a_{1}$ prefer karti hai (80% probability), jo Q-values ke saath align karta hai.

Optimal Policy and Optimal Value Functions

Key insight: Kisi bhi MDP ke liye, kam se kam ek optimal policy $π^{*}$ exist karti hai (unique nahi bhi ho sakti). Ek baar $Q^{*} (s, a)$ pata chal jaye, optimal policy hai: $π^{*} (s) = ar g max_{a} Q^{*} (s, a)$

Kyun? Agar tum hamesha sabse high Q-value wala action choose karo, tum optimal Q-function ke saath greedy ho rahe ho, jo ki optimal policy hai.

$V^{*}$ ke liye Bellman Optimality Equation:

$V^{*} (s) = max_{a} \sum_{s^{'}, r} p (s^{'}, r ∣ s, a) [r + γ V^{*} (s^{'})]$

Derivation:

Optimal value = best possible expected return
"Best possible" matlab har step par best action choose karna
Toh: $V^{*} (s) = max_{a} Q^{*} (s, a)$
$Q^{*}$ ke liye Bellman equation substitute karo

$Q^{*}$ ke liye Bellman Optimality Equation:

$Q^{*} (s, a) = \sum_{s^{'}, r} p (s^{'}, r ∣ s, a) [r + γ max_{a^{'}} Q^{*} (s^{'}, a^{'})]$

Max kyun? Action $a$ leke $s^{'}$ pahunchne ke baad, tum $s^{'}$ se optimally act karte ho, matlab woh action lo jo $Q^{*} (s^{'}, a^{'})$ maximize kare.

Worked example Bellman Optimality in Action

Setup: 3-state chain

States: ${A, B, C}$
$A$ se: action "right" → $B$ jao, reward +0
$B$ se: action "right" → $C$ jao, reward +1
$C$ : terminal state, koi actions nahi, reward hamesha 0
Discount $γ = 1$ (koi discounting nahi, simplicity ke liye)

Dhundho: $V^{*} (A)$ , $V^{*} (B)$ , $V^{*} (C)$

Solution:

Terminal state se backward work karo: $V^{*} (C) = 0$ Kyun? Terminal state, koi future rewards nahi.

$B$ se: $V^{*} (B) = max_{a} [r + γ V^{*} (s^{'})] = 1 + 1 \cdot V^{*} (C) = 1 + 0 = 1$ Kyun? Sirf ek action available hai (right jao), immediate reward +1, phir $C$ pahuncho jiska value 0 hai.

$A$ se: $V^{*} (A) = max_{a} [r + γ V^{*} (s^{'})] = 0 + 1 \cdot V^{*} (B) = 0 + 1 = 1$

Kyun? $B$ ki taraf right jao, koi immediate reward nahi, lekin $B$ ki value 1 hai.

Interpretation: Optimal policy hai $A$ aur $B$ dono se "right jao." Agent ko total reward +1 milta hai (jab $B$ se $C$ transition hota hai).

Common Mistakes

Active Recall

Recall Ek 12-saal ke bachche ko samjhao

Socho tum ek video game khel rahe ho. Tum sabse high score laana chahte ho.

Policy tumhara game plan hai—jaise "jab main ek enemy dekhoon, toh mujhe jump karna chahiye" ya "jab main ek coin dhundoon, toh main use pick up karta hoon." Yeh woh rules hain jo tum follow karte ho har situation mein decide karne ke liye kya karna hai.

Value function ek score predictor jaisa hai. Yeh tumhe batata hai, "Agar main game mein is jagah par hoon, apna game plan follow karte hue, toh game khatam hone tak main kitne points lunга?"

Do types hain:

V(s): "Yeh jagah kitni achi hai?" (Bas is par based ki tum kahan ho.)
Q(s,a): "Is jagah par yeh specific action karna kitna acha hai?" (Jaise, "Agar main abhi jump karoon, toh main kitne points lunga?")

Sabse cool part? Tum Q use karke apna game plan improve kar sakte ho: bas hamesha woh action karo jiska Q-value sabse zyada ho!

Connections 5.1.01-Introduction-to-RL — Policies aur value functions RL problem ko formalize karte hain

5.1.02-Markov-Decision-Processes — MDPs policies define karne aur values compute karne ka mathematical framework provide karte hain
5.1.04-Bellman-Equations — Bellman equations V aur Q ke liye recursive relationships hain
5.2.01-Policy-Iteration — Value functions use karke policies ko iteratively improve karta hai
5.2.02-Value-Iteration — Directly optimal value function compute karta hai, phir optimal policy extract karta hai
5.3.01-Q-Learning — Model-free algorithm jo experience se seedha Q* seekhta hai
5.3.03-PolicyGradient-Methods — Parameterized policies $π_{θ}$ ko directly optimize karta hai

#flashcards/ai-ml

RL mein policy kya hoti hai? :: States se actions ki taraf (ya actions ke distributions ki taraf) ek mapping. Yeh agent ke behavior ko define karti hai. Deterministic: $a = π (s)$ ; Stochastic: $π (a ∣ s)$ .

State-value function

V^{π} (s)

kya hai?

Expected cumulative discounted reward jo state

s

se start karke, policy

π

follow karte hue milta hai:

V^{π} (s) = E_{π} [\sum_{k = 0}^{\infty} γ^{k} R_{t + k + 1} ∣ S_{t} = s]

Action-value function

Q^{π} (s, a)

kya hai?

Expected cumulative discounted reward jo state

s

se start karke, action

a

leke, phir policy

π

follow karte hue milta hai:

Q^{π} (s, a) = E_{π} [\sum_{k = 0}^{\infty} γ^{k} R_{t + k + 1} ∣ S_{t} = s, A_{t} = a]

V aur Q ka aapas mein kya relation hai?

V^{π} (s) = \sum_{a} π (a ∣ s) Q^{π} (s, a)

. State ki value policy se sample kiye gaye actions par expected Q-value hoti hai.

V^{π}

ke liye Bellman expectation equation kya hai?

V^{π} (s) = \sum_{a} π (a ∣ s) \sum_{s^{'}, r} p (s^{'}, r ∣ s, a) [r + γ V^{π} (s^{'})]

. Yeh recursively ek state ki value ko successor states ki values se relate karta hai.

Q^{π}

ke liye Bellman expectation equation kya hai?

Q^{π} (s, a) = \sum_{s^{'}, r} p (s^{'}, r ∣ s, a) [r + γ \sum_{a^{'}} π (a^{'} ∣ s^{'}) Q^{π} (s^{'}, a^{'})]

. Yeh Q ko immediate reward plus discounted future Q-values mein decompose karta hai.

Optimal policy

π^{*}

kya hoti hai?

Woh policy jo saare states ke liye value function maximize kare:

V^{π^{*}} (s) \geq V^{π} (s)

saare

s

aur saari policies

π

ke liye.

Q^{*} (s, a)

se optimal policy kaise extract karte hain?

π^{*} (s) = ar g max_{a} Q^{*} (s, a)

. Har state mein sabse high optimal Q-value wala action choose karo.

V^{*}

ke liye Bellman optimality equation kya hai?

V^{*} (s) = max_{a} \sum_{s^{'}, r} p (s^{'}, r ∣ s, a) [r + γ V^{*} (s^{'})]

. Optimal value actions ke maximum par expected immediate reward plus discounted successor value hai.

Q^{*}

ke liye Bellman optimality equation kya hai?

Q^{*} (s, a) = \sum_{s^{'}, r} p (s^{'}, r ∣ s, a) [r + γ max_{a^{'}} Q^{*} (s^{'}, a^{'})]

. Optimal Q-value expected immediate reward plus next state mein discounted maximum Q-value hai.

Discount factor

γ < 1

kyun use karte hain?

Infinite sums ka convergence ensure karta hai (geometric series), future ke baare mein uncertainty model karta hai, aur time preference represent karta hai. Iske bina, positive rewards wale infinite-horizon problems mein infinite value hoti hai.

V^{π} (s)

aur

V^{*} (s)

mein kya difference hai?

V^{π} (s)

ek specific policy

π

ke under value hai (

π

ke according actions par average).

V^{*} (s)

optimal value hai (saari policies par maximum, ya equivalently, actions par max).