Agent, environment, state, action, reward — Hinglish notes

5.1.1 · AI-ML › Reinforcement Learning Foundations

Overview

Agent-environment interaction loop reinforcement learning ki foundation hai. Supervised learning mein hume bataya jaata hai ki sahi answer kya hai, lekin RL agents ko achha behavior trial and error se discover karna padta hai — unhe sirf itna feedback (rewards) milta hai ki unke actions kitne achhe the.

Figure — Agent, environment, state, action, reward

The Five Components

Key insight: Agent mein "brain" hota hai — policy (action-choosing strategy) aur possibly ek model ki duniya kaise kaam karti hai.

Key insight: Environment boundary ek design choice hai. Ek robot ke liye, hum robot ki body ko environment mein include kar sakte hain aur sirf control algorithm ko agent maan sakte hain. Ya motor controllers ko agent mein rakh sakte hain aur sirf baaki duniya ko environment mein.

Ye kyun matter karta hai: Fully observable environment mein (jaise chess), agent sahi state dekhta hai. Partially observable environments mein (jaise poker ya limited sensors wale robot), agent ko incomplete information se decisions lene padte hain.

Actions kyun first-class hain: Agent SIRF apne actions control karta hai. Woh directly state ya reward control nahi kar sakta — wo interaction se emerge hote hain.

Critical insight: Rewards SIRF ek hi tarika hain goal communicate karne ka. Agar tum chahte ho ki RL agent aage chale, toh forward distance reward karo. Agar galti se "na girna" reward kiya, toh agent lait jaana seekh sakta hai!

The Interaction Loop: Deriving the Dynamics

Chalo formalize karte hain ki ye components time ke saath kaise interact karte hain.

Step 1: Initial state $t = 0$ par, environment state $s_{0}$ mein hai (kisi initial distribution $p (s_{0})$ se drawn).

Step 2: Agent observes Agent observation $o_{0}$ receive karta hai (fully observable case mein, $o_{0} = s_{0}$ ).

Step 3: Agent acts Agent ki policy $π$ states/observations ko actions pe map karti hai. Ye ho sakta hai:

Deterministic: $a_{t} = π (s_{t})$ — same state hamesha same action deta hai
Stochastic: $a_{t} \sim π (a ∣ s_{t})$ — probability distribution se action sample karo

Step 4: Environment responds Environment action $a_{0}$ receive karta hai aur:

Naye state mein transition karta hai: $s_{1} \sim P (s^{'} ∣ s_{0}, a_{0})$ jahan $P$ state transition probability hai
Reward emit karta hai: $r_{0} = R (s_{0}, a_{0}, s_{1})$ ya $r_{0} \sim R (s_{0}, a_{0})$ agar stochastic hai

Step 5: Loop repeats Isse ek trajectory produce hoti hai (episode ya rollout bhi kaha jaata hai): $s_{0}, a_{0}, r_{0}, s_{1}, a_{1}, r_{1}, s_{2}, a_{2}, r_{2}, \dots$

Ye formulation kyun? Ye ek Markov Decision Process (MDP) hai. "Markov" part ka matlab hai: current state $s_{t}$ diya ho toh future past se independent hai. Formally: $P (s_{t + 1}, r_{t} ∣ s_{t}, a_{t}, s_{t - 1}, a_{t - 1}, \dots) = P (s_{t + 1}, r_{t} ∣ s_{t}, a_{t})$

Isse hum decisions sirf current state ke basis par le sakte hain, poori history ke basis par nahi.

Finite horizon (episode time $T$ par khatam hota hai): $G_{t} = r_{t} + r_{t + 1} + r_{t + 2} + \dots + r_{T} = \sum_{k = 0}^{T - t} r_{t + k}$

Kyun? Simple sum tab kaam karta hai jab episodes ka ek clear end ho (game over, task complete).

Infinite horizon with discounting: $G_{t} = r_{t} + γ r_{t + 1} + γ^{2} r_{t + 2} + \dots = \sum_{k = 0}^{\infty} γ^{k} r_{t + k}$

jahan $γ \in [0, 1)$ discount factor hai.

Discount kyun?

Mathematical: Infinite sum ko converge karta hai (agar rewards bounded hain)
Practical: Ye reflect karta hai ki near-term rewards distant rewards se zyada certain/valuable hain
Control: $γ$ 0 ke close = myopic (greedy), $γ$ 1 ke close = far-sighted

Ye kyun converge hota hai, derivation: Maano rewards bounded hain: $∣ r_{t} ∣ \leq R_{ma x}$ sab $t$ ke liye. $∣ G_{t} ∣ = \sum_{k = 0}^{\infty} γ^{k} r_{t + k} \leq \sum_{k = 0}^{\infty} γ^{k} R_{ma x} = R_{ma x} \sum_{k = 0}^{\infty} γ^{k}$

Geometric series ka sum hai: $\sum_{k = 0}^{\infty} γ^{k} = \frac{1}{1 - γ} for ∣ γ ∣ < 1$

Isliye: $∣ G_{t} ∣ \leq \frac{R _{ma x}}{1 - γ}$ jo finite hai. ✓

Recursive form (bootstrapping): $G_{t} = r_{t} + γ (r_{t + 1} + γ r_{t + 2} + \dots) = r_{t} + γ G_{t + 1}$

Ye recursion Bellman equation ka beej hai.

Trajectory example:

t=0: s₀=(1,1), agent observes (1,1), chooses a₀=North
     Environment: valid move, s₁=(1,2), r₀=-1

t=1: s₁=(1,2), agent chooses a₁=East
     Environment: valid move, s₂=(2,2), r₁=-1

t=2: s₂=(2,2), agent chooses a₂=North  
     Environment: hits wall, s₃=(2,2), r₂=-5

t=3: s₃=(2,2), agent chooses a₃=East
     Environment: valid move, s₄=(3,2), r₃=-1

... continues until reaching goal at (5,5)

t=12: s₁₂=(5,5), r₁₂=+10, episode ends

Return calculation ( $γ = 0.9$ ke saath): $G_{0} = - 1 + 0.9 (- 1) + 0. 9^{2} (- 5) + \dots + 0. 9^{12} (+ 10)$

Step penalty kyun? Iske bina ( $r = 0$ non-goal states ke liye), agent ko jaldi goal reach karne ki koi zaroorat nahi. Woh hamesha ke liye ghoom sakta hai aur kabhi achhi policy nahi seekhega.

Ye reward structure kyun?

Pehla term: Negative squared error → target se deviation penalize karta hai (comfortable temperature)
Doosra term: Energy use par small penalty → efficiency encourage karta hai
Dono terms negative hain → reward maximize karna matlab deviations aur energy minimize karna

Trajectory step:

t=0: s₀=(18°C, 10°C), agent chooses a₀=0.8 (80% heater)
     Environment physics: Q = heater_power × max_heating
                Heat_loss = k × (T_inside - T_outside)
                T_new = T_old + (Q - Heat_loss) × dt
     Result: s₁=(19.5°C, 10°C), r₀=-(19.5-21)² - 0.01×0.8² = -2.256

Continuous actions kyun? Temperature control naturally continuous scale par kaam karta hai. {off, low, medium, high} mein discretize karne se precision kho jaati.

Common Mistakes

Ye sahi kyun lagta hai: Rewards batate hain kya achha hai, toh har reward maximize karna logical lagta hai.

Fix: Agent ko cumulative discounted reward (return) maximize karna hota hai, na ki immediate reward. Example: Chess mein ek piece sacrifice karna (negative immediate reward) checkmate set up karne ke liye (bada future reward) optimal hai.

Formal: Agent ek policy $π^{*}$ seekhta hai jaise ki: $π^{*} = ar g max_{π} E_{τ \sim π} [\sum_{t = 0}^{\infty} γ^{t} r_{t}]$ Na ki: $π = ar g max_{π} E [r_{t}]$ har $t$ par independently.

Ye sahi kyun lagta hai: Losses aur rewards dono performance measure karte hain, aur hum ML mein minimization ke aadat hain.

Fix: RL rewards maximize karne ke liye design hote hain. Agar tumhare task mein naturally loss hai (tracking error, energy waste), toh define karo $r = - loss$ taaki reward maximize karna matlab loss minimize karna ho.

Example: Goal se distance $d$ ke liye, use karo $r = - d$ na ki $r = d$ .

Ye sahi kyun lagta hai: Simple states = smaller state space = faster learning.

Fix: Agar tum future predict karne ke liye zaroori information chod dete ho, toh Markov property toot jaati hai. Agent same state dekhega lekin hidden history ki wajah se alag outcomes experience karega, jo learning impossible bana deta hai.

Example: Pong mein, ek single frame ball velocity nahi dikhata. Tumhe 2+ frames chahiye ya explicitly state mein velocity include karni padegi. Warna, agent predict nahi kar sakta ball kahan jaayegi.

Ya ek feedback loop imagine karo: Agent → Action → Environment → State+Reward → Agent (loop close hota hai)

Recall Ek 12-Saal ke Bacche ko Samjhao

Socho tum ek naya video game seekh rahe ho, lekin controls labeled nahi hain. Tum buttons dabaate ho aur dekhte ho kya hota hai. Jab tum ek coin uthate ho, game tumhe points deta hai (woh reward hai). Jab tum ek hole mein girte ho, toh points kho dete ho (negative reward).

Tum (agent) screen dekh sakte ho (state) — tum kahan ho, enemies kahan hain. Tum choose karte ho kaun sa button dabao (action). Game ki duniya (environment) tumhare button ke hisaab se update hoti hai: tumhara character move karta hai, enemies move karti hain, aur decide hota hai tumne kitne points kamaaye.

Pehle tum random buttons try karte ho. Lekin time ke saath tum patterns notice karte ho: "Jab main cliff ke paas hoon aur right dabaata hoon, toh main girta hoon aur points kho deta hoon. Jab main cliff ke upar jump karta hoon, toh safely paar ho jaata hoon." Tum seekh rahe ho kaun se actions kaun si situations mein kaam karte hain. RL agents exactly yahi karte hain, sirf button mashing ki jagah math ke saath!

"Return" waise hai jaise pure game ka total score. Tum sirf sabse paas ka coin pakadna nahi chahte (immediate reward) — kabhi kabhi ek coin skip karna smarter hota hai khatre se bachne ke liye, taaki end mein total score better ho.

Connections

Markov Decision Process (MDP) — formal mathematical framework
Policy and Value Functions — agents apni strategy kaise encode karte hain
Bellman Equations — optimal returns ke liye recursive relationship
Exploration vs Exploitation — agents naye actions try karne aur jaane-maane achhe actions use karne mein balance kaise karte hain
Partial Observability (POMDP) — jab agent poora state nahi dekh sakta
Multi-Armed Bandits — simplified RL sirf ek state ke saath
Model-Free vs Model-Based RL — agent environment dynamics seekhta hai ya nahi

#flashcards/ai-ml

Reinforcement learning ke paanch core components kya hain?

Agent (learner/decision-maker), Environment (duniya), State (situation ka description), Action (agent ki choices), Reward (feedback signal)

Environment state aur observation mein kya farq hai?

Environment state duniya ki complete, sach mein state hoti hai. Observation (agent state) woh hai jo agent actually perceive karta hai, jo partial ya noisy ho sakti hai. Fully observable environments mein dono barabar hoti hain.

RL mein Markov property kya hai?

Current state diya ho toh future past se independent hai: P(s_{t+1}, r_t | s_t, a_t, history) = P(s_{t+1}, r_t | s_t, a_t). Iska matlab hai ki current state mein future predict karne ke liye saari zaroori information hoti hai.

Return G_t kya hai aur hum ise kyun use karte hain?

Return cumulative discounted reward hai: G_t = Σ γ^k r_{t+k}. Hum ise isliye use karte hain kyunki agent ka goal total long-term reward maximize karna hai, na ki sirf turant reward. Discounting (γ) near-term rewards ko distant ones se zyada valuable banata hai.

Hum future rewards discount kyun karte hain (γ < 1 use kyun karte hain)?

Teen reasons: (1) Mathematical — infinite sums converge hote hain, (2) Practical — near-term rewards zyada certain hote hain, (3) Control — short-term vs long-term focus tune karne deta hai. γ 0 ke close = greedy, γ 1 ke close = far-sighted.

RL mein trajectory kya hai?

Agent-environment interaction se produce hone wala states, actions, aur rewards ka sequence: s₀, a₀, r₀, s₁, a₁, r₁, s₂, a₂, r₂, ... Episode ya rollout bhi kaha jaata hai.

Discrete aur continuous actions mein kya farq hai?

Discrete actions ek finite set se hoti hain (jaise {left, right, jump}). Continuous actions ek continuous range se hoti hain (jaise steering angle ∈ [-30°, +30°]). Alag action spaces ke liye alag RL algorithms chahiye.

Reward task specify karne ka akela tarika kyun hai?

Reward function SIRF ek signal hai jo agent ko batata hai ki hum kya chahte hain. Agent hamare mann nahi padh sakta — woh jo bhi reward define kiya hai use maximize karega. Agar reward hamare sache goal se align nahi karta, toh agent galat behavior seekhta hai (reward hacking).

Har step par immediate reward maximize karne mein kya galat hai?

Ye myopic/greedy behavior hai. Optimal RL ke liye cumulative discounted return maximize karna zaroori hai, jisme better long-term outcomes ke liye low/negative immediate rewards accept karne pad sakte hain (jaise checkmate set up karne ke liye chess piece sacrifice karna).

States ko Markov property kyun satisfy karni chahiye?

Agar zaroori information history mein chupi ho, toh agent same state dekhta hai lekin us history ki wajah se alag outcomes experience karta hai. Isse learning impossible ho jaati hai kyunki state mein future predict karne ke liye kaafi information nahi hoti.

Return ki recursive form kya hai?

G_t = r_t + γ·G_{t+1}. Time t par return equals immediate reward plus next step se discounted return. Ye recursion seedha Bellman equation ki taraf le jaata hai.